Split the input JSON to multiple files?

[robbrad]

Hi team, I'm considering the merge conflicts on input.json (and the constant reordering for alphabetical order — apologies for my OCD). What are your thoughts on splitting the input.json into multiple files — one for each council?

If we do this, we could also contemplate moving to a folder per council in the test folder, making it easy to copy a new one and repeat the process.

For instance:
Council_test_folder ->

Schema file
Input file
Output file

Furthermore, it would be more systematic to have the actual class file in the folder too, so that each council becomes a self-contained folder with functional code and tests, etc. This change is simply for human convenience and reduces the effort of creating the right files each time.

What do you think? @OliverCullimore / @dp247

[OliverCullimore]

@robbrad I was thinking of making the dev mode reorder the input.json on each run and also the feature file if was possible also but yeah that's another solution

I was also thinking ref the schema files, now were using a common output format, do we need one for each council or could we have a single one for all that's more thorough (not sure if we can specify regex patterns in json schema to check the date format is correct and also if we can set a min occurrence of 1 collection to be valid?)

[robbrad]

You are right - a common schema file would be the utopia and also really what JSON schemas were intented to be used.

It would add more resiliant too - meaning we dont need to rely on a conversion process

[robbrad]

One thing to call out is - the schema for a council is a great way to see if they changed their site or its broken - we should be concious that the schema doesnt give a false postive of success

[OliverCullimore]

Although if we did split out input.json we'd have to rethink how the custom component gets it's councils list as currently it pulls that single file, we'd need it to run off the feature file instead perhaps as I'm not sure github raw urls will do directory listing for us to grab the folder names?

One other solution could be making it so dev mode adds to the input.json and feature file if an entry doesn't exist therefore you just have to add the council Python file and run collect data with the dev mode flag? We could even populate the input.json entry based on the passed flags.

[dp247]

That sounds like my utopia lol

[OliverCullimore]

@dp247 check out #409 and #415

Only thing it doesn't do is update the feature file as I'm not too sure how to update that one in the same sort of way