I haven't updated this blog in a while, but here's what I've been up to over the last couple months with my project.
I debugged the SQL issues, which was a time-consuming process, but after a while I finally realized that the old data that was still in the permanent database could be causing the issue. But after relocating to a fresh database, for some reason 1) the error was still happening and 2) there was still data from February in the table deep into the processing stage where the error happens. After opening the new MetasysLog database using database viewing software and sorting by timestamp, it was clear that everything in that database is from September. This meant the program was still feeding in old data from another database. It turned out that although I had been updating DailyDatabase, DailyTempDatabase, DailyCarbonDatabase, and FilteredT3Database (all from Task 3), because there were errors on Task 4, I never reached the end of Task 4 where I was able to clear the databases. For this reason, the old data from February was still in the databases. As a result of discovering the bug, I created and ran a separate file to clear the databases.
Next, after clearing the databases, when reading from DailyDatabase, I got a
ValueError: invalid literal for int() with base 10: b'=\x00\x00\x00\x00\x00\x00\x00'`. In order to solve this, I wrote a function fix_bytes() and applied it to the columns that were producing the byte values (all the Intervals Too ____ columns) just before writing to SQL. Then, in posttask4.py, I modified the convert_to_int function to be compatible with floats as well. This made everything run smoothly!
After fixing the data type conversion issues, I was able to run everything with no errors! However, I then opened up the resulting spreadsheets to find that almost every room was flagged as having a potential CO2 sensor issue... and they all had values at around 60-80 ppm... which were all the same numbers as their temperatures in Celsius... I was not sure where the program swapped out the CO2 values for temperature ones. After checking the databases at each part of the program, I was able to zone in on the problem: there was a basic copy error within convert_metasys_data.py which replaced all of the CO2 values with all of the temperature values while running transformations. After fixing this and messing a little bit more with conversions, I was able to fix the issue!
Now that the application seems to have cleared basic testing and runs without obvious errors, I have to check the details.
I've already noticed one potential issue: some rooms have moments with both low co2 and high co2, but they're only found within the low co2 spreadsheet, while other rooms are the same way and can be found in both. This inconsistency should be further examined, and I might create a smaller testing file in order to catch more.