Hello,
I’ve been evaluating a few functions and properties around timeseries new features: the test case is to get data from a .csv typical test file downloaded from french DSO enedis for residential power meter “Linky”, as illustrated below.
The data range is about 2 years, of hourly power consumption records (some irregular sample times here and there).
Identifiant PRM;Type de donnees;Date de debut;Date de fin;Grandeur physique;Grandeur metier;Etape metier;Unite;Pas en minutes
12345678909876;Courbe de charge;11/01/2022;01/01/2024;Energie active;Consommation;Comptage Brut;W;
Horodate;Valeur
2022-01-11T01:00:00+01:00;278
2022-01-11T02:00:00+01:00;1462
2022-01-11T03:00:00+01:00;1501
2022-01-11T04:00:00+01:00;601
2022-01-11T05:00:00+01:00;344
2022-01-11T06:00:00+01:00;365
2022-01-11T07:00:00+01:00;594
2022-01-11T08:00:00+01:00;518
2022-01-11T09:00:00+01:00;412
2022-01-11T10:00:00+01:00;378
2022-01-11T11:00:00+01:00;520
2022-01-11T12:00:00+01:00;431
2022-01-11T13:00:00+01:00;474
2022-01-11T14:00:00+01:00;418
2022-01-11T15:00:00+01:00;395
2022-01-11T16:00:00+01:00;289
2022-01-11T17:00:00+01:00;264
2022-01-11T18:00:00+01:00;372
2022-01-11T19:00:00+01:00;309
2022-01-11T20:00:00+01:00;404
I first tried to detect the import option automatically (as readtimeseries() with no options ) but it failed because of the header structure, with some irregular introductive lines in the beginning of files until the real regular data:
dummyfile= "TMPDIR\Enedis_Conso_Heure_20220111-20240101_12345678909876.csv"
opts= detectImportOptions(dummyfile)
it fails :
Attention : Une incohérence a été trouvée dans les colonnes. À la ligne 2, 2 colonnes ont été trouvées, alors que la précédente en avait 9.
à la ligne 64 de la fonction detectImportOptions ( C:\Program Files\scilab-2024.0.0\modules\spreadsheet\macros\detectImportOptions.sci ligne 75 )
csvTextScan: can not read file, error in the column structure
then I created manually import structure : several attempts required to properly set the details of the fields ‘header’ and ‘inputFormat’ (help page could be refined with more exotic test cases here I guess. or improvement of autodetect header file) :
opts= struct("variableNames", ["Horodate","Valeur"],"variableTypes", ["datetime","double"],"delimiter", ";","datalines", [4,33448],"header", ["Identifiant PRM;Type de donnees;Date de debut;Date de fin;Grandeur physique;Grandeur metier;Etape metier;Unite;Pas en minutes"
"12345678909876;Courbe de charge;11/01/2022;01/01/2024;Energie active;Consommation;Comptage Brut;W;"
"Horodate;Valeur"],"inputFormat","yyyy-MM-ddTHH:mm:ss+","emptyCol",[])
ts= readtimeseries(dummyfile,opts)
which ran successully, except the current limitation (warning is properly issued) to manage UTC and daylight saving time information in datetime : then everything behind the ‘+’ in inputformat, the ‘+01:00 or +02:00’ information is not used to obtain true local time : this may be also indicated in the help page.
ATTENTION : UTC/GMT format is not managed. The result does not take it into account.
ts =
…
33445x1 timeseries
Horodate Valeur
… ___________________ ______
…
2022-01-11 01:00:00 278
2022-01-11 02:00:00 1462
2022-01-11 03:00:00 1501
… …
2023-12-31 23:00:00 144
2023-12-31 23:30:00 126
2024-01-01 00:00:00 152
Then i tried the timeseries plot feature
stackedplot(ts)
it’s fine to get basic outlook of the whole data but the lack of callback function to set automatically the x_ticks with properly formatted datetime information according to the level of zoom prevent any real practical interpretation of this plot:
Then I suggest that the Simple date and time plotting on x-axis is still an issue to work on in the next Scilab releases.
Thank you for the discussion and the developments already performed!
David