feat(upload): improve SINAN upload; add helpers, validations & views to upload large files

[luabida]

No description provided.

[luabida]

I don't know if its supposed to happen this way, but this dropna usually drops the entire chunk, few rows don't have None in some of the required columns

Edit: found the issue, the dropna is correct since required_cols can't contain None values, I was incorrectly replacing a string by None

[fccoelho]

Is this DROPNA necessary for the conversion to Parquet? If not, we should retain all rows independently of the fact that they may contain NAs. This chunk_parquet_file should not make decisions about data quality. It should already receive sanitized data.

[luabida]

I thought that the parquet file would contain the raw data (the DBF converted to parquet, for instance) and the inserted rows of that file would be pointed in a separated table..

[fccoelho]

You are right. The sanitization should be done as a separate step.

[luabida]

@fccoelho I'll merge this PR the way it is right now so the SINAN upload can be used on production, but its still missing the file overview (merely visual) and the task to convert the dbf to parquet (until we resolve how the data should be included on the parquet file). Can you review it again please?

[fccoelho]

@luabida I think that the Parquet should have the same exact content of the DBF(as you mentioned above), so that we don't need to keep the DBFs.
But we should have separate diagnostic step that can report problems in the DBF, before it is inserted in the production database. Simple things like checking it has all columns we need, counting the number of NAs per column, etc.

[fccoelho]

Looks Good.