Add UHDM frontend support for Yosys using Surelog #202

kamilrakoczy · 2020-12-11T10:18:58Z

This PR adds UHDM frontend for yosys.

kgugala · 2020-12-11T11:29:22Z

Some context - this allows synthesis of SystemVerilog designs in Yosys using Surelog parser. Surelog passes information about the design using UHDM.

Here https://github.com/alainmarcel/uhdm-integration/pull/97/checks?check_run_id=1536854237 you can find an example Ibex build. The design builds fine and can be P&Rd with e.g. Vivado giving working bitstream.

imphil · 2020-12-18T15:53:08Z

Thanks! I'll give this a try in the new year with Ibex.

kgugala · 2020-12-22T10:35:28Z

I've rebased this on top of master

olofk · 2020-12-22T18:17:03Z

edalize/surelog.py

+        if api_ver == 0:
+            return {'description' : "Surelog",
+                    'lists' : [
+                        {'name' : 'library_files',


This needs som explanation. I get the feeling it is verilog "libraries", i.e. a directory with files that have the same filename as the modules inside, or is it something else? If it is verilog libraries I wonder if this is really the method we prefer. It should be possible to specify these as ordinary source files instead.

Generally when files are added as tool_options we must be very careful about thinking how the user will pass them into Edalize. In most cases it's better to add a new file type if we must

Surelog requires access to definition of all modules to match the usage with the model specification (without this, Surelog will be missing parameter information for those modules).

This parameter allows to pass such files as a libraries (more information available in: https://github.com/alainmarcel/Surelog/issues/893).

I tried to pass them as a logical_name file_type , but when using edalize with fusesoc, files can only be specified relative to the .core location (and this files are usually stored outside of that location). If there is better way to pass this files that will work with fusesoc I will change this.

Example of such library file: https://github.com/antmicro/yosys/blob/uhdm-yosys/techlibs/xilinx/cells_xtra_surelog.v

I agree with Olof, the library_files directive here looks wrong and probably needs to go -- I'm just not quite sure yet how. To get us closer to that answer: Are these libraries needed when a user actually uses a Xilinx primitive in their code, or are they always needed when the synthesis tool decides to infer a primitive?

They are only needed, when a user uses a Xilinx primitive in their code.

Potentially any vendor primitives (this is not limited to Xilinx)

OK, a follow-up then: How are those libraries distributed? Are they part of Surelog, or of (the modified version of) Yosys? Or is the user expected to provide those files themselves?

They are distributed as a part of yosys or symbiflow_toolchain (in case of building using symbiflow_toolchain instead of vivado).

olofk · 2020-12-22T18:18:35Z

edalize/surelog.py

+        (src_files, incdirs) = self._get_fileset_files()
+        file_list = []
+        for f in src_files:
+            if f.file_type in ['verilogSource']:


This will ignore e.g. verilogSource-95 types. Use startswith instead

olofk · 2020-12-22T18:21:17Z

edalize/surelog.py

+        for key, value in self.vlogparam.items():
+            verilog_params_command = verilog_params_command + " -P" + key + '=' + value
+
+        include_files = self.tool_options.get('include_files', [])


Where do I find these include_files?

It was unused leftover, I removed this.
I'm using incdirs that is returned by self._get_fileset_files()

imphil

A first set of comments form my side. All of the current Yosys backend work is quite FPGA-specific where it doesn't need (or should) be; I need to have a closer look at that first, because it seems that this PR pushes into that direction even more.

I'm also not really happy with the fact that we now effectively have a tree of options (vivado -> vivado/yosys -> surelog/plain-yosys) all slightly intermixed. Maybe we need to just start calling different tool "mixes" and patched versions differently, e.g. "vivado-yosys" or "vivado-synplify" or "yosys-surelog", as long as the tool (vendor) doesn't integrate those options into a single product.

imphil · 2021-01-09T20:31:55Z

edalize/surelog.py

+        if api_ver == 0:
+            return {'description' : "Surelog",
+                    'lists' : [
+                        {'name' : 'library_files',


I agree with Olof, the library_files directive here looks wrong and probably needs to go -- I'm just not quite sure yet how. To get us closer to that answer: Are these libraries needed when a user actually uses a Xilinx primitive in their code, or are they always needed when the synthesis tool decides to infer a primitive?

imphil · 2021-01-09T20:32:59Z

edalize/templates/surelog/surelog-makefile.j2

@@ -0,0 +1,27 @@
+#Auto generated by Edalize
+SHELL = bash


Is that needed?

Apparently not, removed

imphil · 2021-01-09T20:37:22Z

edalize/surelog.py

+        template_vars = {
+                'top'                       : self.toplevel,
+                'name'                      : self.name,
+                'sources'                   : ' '.join(file_list),


According to the documentation, surelog differentiates between Verilog and SystemVerilog parsing. Currently, the call always forces Surelog to operate in SystemVerilog mode and then passes files as "Verilog". If Surelog makes that difference, we should honor the file type passed in FuseSoC and pass on files with verilogSource and systemVerilogSource differently. (If Surelog treats those files the same internally, maybe its command line options should be cleaned up.)

-sverilog Forces all files to be parsed as SystemVerilog files -sv <file> Forces the following file to be parsed as SystemVerilog file

I noticed we don't force SystemVerilog mode anymore. By default we let Surelog handle Verilog and SV internally.

mithro · 2021-01-10T22:13:40Z

@imphil - I recently tried to again map out the how the different parts of the ecosystem connect to each other.

You can find it at https://docs.google.com/drawings/d/1ocoKZFu2gJodAgBku4yTePjdXSrl2tBg95czmqivvY8/edit

I really think we should move edalize towards an approach were it understands things like "language frontend" plugins and also the different stages like "synthesis" verse "place and route". It is perfectly reasonable to want to use (Surelog+UDHM for SV)+(GHDL for VHDL)->Yosys->Vivado for a mixed synthesis flow. To make things even more complicated there is overlap between the language frontends and the languages they support, for example;

all of Yosys native, Verific, Surelog+UHDM should in theory support native Verilog, while
both Surelog+UHDM and Verific support SystemVerilog, and
both GHDL and Verific support VHDL.

Maybe the Yosys synthesis tool could provide a list of "language frontends" and each "language frontend" can register for which file types it supports?

Still unsure how you handle things like Yosys plugins for optimization like abc or lsoracle...

tjurtsch · 2021-05-14T10:04:41Z

library_files directive was removed. We define "arch" instead and use "/cells_bb.v" in yosys-config call in Makefile.

Add EDALIZE_LAUNCHER to verilator template

olofk · 2021-05-20T07:22:10Z

Hi,

Just wanted to say that I'm intending to review this PR next but just haven't found the time yet. Soon come!

olofk · 2021-05-21T13:20:26Z

FYI, this will take a bit more time. The reason now is that I realized that this addition will be a lot more manageable if we first can do some refactoring that is needed anyway for Edalize (Slight Return). I filed a tracking issue to make this a bit more visible #243

olofk · 2021-05-24T20:15:28Z

@kamilrakoczy @kgugala @kamilrakoczy @mithro It would be great if I could get some help from you. As mentioned in #243 I need to do some refactoring first before I can merge this one. What I need help with is to a) check that the current changes on the https://github.com/olofk/edalize/tree/esr1 branch didn't break anything (I think fpga-tool-perf is the main user here) and b) get help converting the rest of the backends in the way that I have done with the three first ones. So if anyone time to help that would help me unblock this one faster

kgugala · 2021-05-25T15:58:11Z

@olofk I must have missed the notification from this one. I'll find somebody to push this further

olofk · 2021-05-25T20:27:55Z

Thanks @kgugala! I pushed a bunch of more patches now to heopfully make it clearer what I'm trying to achieve

…ontend_optins in yosys to reflect that

tgorochowik · 2021-05-28T15:03:38Z

@olofk testing the whole perf tool on the refactored branch will not be easy as the perf tool uses a fork which is still quite far from mainline for now and it's not very simple to rebase it all and test it quickly.

We did however run some limited tests of it, only for icestorm for now, as this part required less effort, we've had to add one fix to your branch to make it pass, here is the patch: antmicro@8edd8c8

The next step for us will be to rebase the work from this PR onto your refactored branch

olofk · 2021-05-31T11:01:23Z

Thanks @tgorochowik . It's exactly these kind of issues I was worried about. Great that you caught it. How would you feel if I merged my esr1 branch to master with that fix applied? I'm a bit worried that it hasn't received much testing, but OTOH it might be quicker to work out issues if it's in master, and perhaps easier for you to add your patches on top of master instead of a branch.

I'm fully aware that the rebase will be painful but a major goal with the refactoring is to make the integration of new tools, features and flows easier so hopefully it will be smoother from now on. Also, I'm done with the big refactoring now. There are probably a couple of cleanups left but other than that all Yosys-using backends are converted. And with that I think we can start landing the surelog and sv2v support piece by piece. I have some ideas for the order of things

Add surelog backend with some changes
- Use commands.add instead of a makefile snippet template
- Consume all (system)verilog and update self.edam with unused files + the uhdm file (See how yosys.py does this with the netlists and unused_files)
Same with sv2v but add generated verilog file(s) to self.edam
Add uhdm support to yosys. Think it should just be another elif in the for f in self.files: loop

After that we can start the integration but there are some details to work out

We want to support multiple frontends, e.g VHDL that goes through ghdl and systemverilog through surelog. For that reason I think we need to make frontends a list instead of a single string.
We need to figure out where to do the integration.
- The GHDL plugin is tied to yosys (but I seem to recall there's also native support in ghdl to produce verilog nowadays?)
- surelog is practically tied to yosys because that's the only tool that reads uhdm but I recall hearing that verilator also is gaining uhdm support
- sv2v can be used by any tool that reads verilog.
- What about slang? Does anything read the output produced by slang?
From the list above there are some frontends that practically should be tied to yosys, but OTOH I think it would be architecturally nicer to allow any backend to potentially use these frontends. So I'm leaning towards having support in each of the backend classes that describe a flow (e.g. symbiflow, trellis, icestorm, vivado...) instead of pushing it into the yosys class. If I have done the refactoring right, it should just be a few lines of code to each backend to add the support.

What do you think?

tgorochowik · 2021-05-31T11:34:49Z

(...) How would you feel if I merged my esr1 branch to master with that fix applied? I'm a bit worried that it hasn't received much testing, but OTOH it might be quicker to work out issues if it's in master, and perhaps easier for you to add your patches on top of master instead of a branch.

We're going to have to rebase everything anyway so I'd say the sooner we do this the better.

Add surelog backend with some changes (...)

We'll start by doing it the way you described and try to come up with some ideas on how to address the details that need to be worked out along the way. This way it will be easier for us to fully understand the current state and propose some ways to continue. Is that okay?

What about slang? Does anything read the output produced by slang?

We don't use slang for anything that might require integration with edalize at the moment, but it certainly can be integrated at a later stage (CC @MikePopoloski)

MikePopoloski · 2021-05-31T11:52:58Z

Right now slang only has two output formats -- JSON and a C++ AST library. I would not recommend JSON as an interchange format, it will be far too bulky and slow for larger projects. The C++ library is ideal from a compilation speed perspective but obviously it requires custom work by each consumer. I'm not against adding UHDM output; I think it would probably be pretty simple (and I believe it uses Cap'n Proto for the database which is a good choice), though when I looked in the past all the docs seemed to be geared towards consumers and I didn't spend much time digging into what was needed on the producer side. This is an area that would be relatively straightforward for someone to contribute to slang if they wanted.

olofk · 2021-05-31T21:23:15Z

(...) How would you feel if I merged my esr1 branch to master with that fix applied? I'm a bit worried that it hasn't received much testing, but OTOH it might be quicker to work out issues if it's in master, and perhaps easier for you to add your patches on top of master instead of a branch.

We're going to have to rebase everything anyway so I'd say the sooner we do this the better.

Great. I agree. It's a big patcheset, but we just had a release also, so this is probably the safest time to merge potentially broken stuff. It's in master now.

Add surelog backend with some changes (...)

We'll start by doing it the way you described and try to come up with some ideas on how to address the details that need to be worked out along the way. This way it will be easier for us to fully understand the current state and propose some ways to continue. Is that okay?

That sounds great to me!

tgorochowik · 2021-07-27T10:51:48Z

Just to provide an update here:
In the end we've decided it is a good moment to start using the UHDM frontend in yosys as a plugin - a the same time we redid the Edalize support to use plugins.

The current state of Edalize (with links to current code) is tracked here: antmicro/yosys#443

The state of the plugin itself is tracked here: antmicro/yosys#442

For the time being we're going to keep it separated, when the plugin is ready and usable we will contribute the edalize changes to this repo (either in this PR or separately in a new one).

olofk · 2021-07-28T22:23:57Z

Thanks for the update @tgorochowik. FYI, I would still be happy to pull sv2v support first if you want to keep the whole patchset smaller.

olofk · 2021-12-20T23:07:00Z

A couple of weeks ago I landed the first patches of the new flow API which supports using surelog as a frontend. Does this do the job so we can close this PR?

kamilrakoczy · 2021-12-21T09:06:42Z

I answered your question about state of the new flow API in: #261.
This PR is superseded by: #261, closing.

imphil self-requested a review December 17, 2020 11:09

kgugala force-pushed the surelog branch from 8aec3ce to 3a49edc Compare December 22, 2020 10:34

olofk reviewed Dec 22, 2020

View reviewed changes

kamilrakoczy requested a review from olofk January 5, 2021 10:38

imphil reviewed Jan 9, 2021

View reviewed changes

kamilrakoczy force-pushed the surelog branch from 07b82c0 to 20f1f27 Compare January 22, 2021 08:18

mtdudek force-pushed the surelog branch from 20f1f27 to 27b18a5 Compare May 12, 2021 12:32

tjurtsch force-pushed the surelog branch from 27b18a5 to e4e92a1 Compare May 14, 2021 08:17

kamilrakoczy and others added 3 commits May 14, 2021 11:55

Add Surelog support

650736f

yosys: enable Surelog frontend to parse SystemVerilog

7896cdf

Make Surelog frontend available to every tool using Yosys

c693144

tjurtsch force-pushed the surelog branch from e4e92a1 to c693144 Compare May 14, 2021 09:55

mtdudek force-pushed the surelog branch from 7b2d912 to 5e3e584 Compare May 19, 2021 16:24

Move verilator templates to templates/verilator directory

bb56326

Add EDALIZE_LAUNCHER to verilator template

mtdudek force-pushed the surelog branch from 7e82910 to bb56326 Compare May 19, 2021 19:02

Add support for EDALIZE_LAUNCHER for surelog makefile

cd43099

mtdudek added 2 commits May 26, 2021 14:12

Add support for sv2v frontend to yosys, changed surelog_options to fr…

e13512a

…ontend_optins in yosys to reflect that

Fix missing edif_opts in yosys script

410f8a4

Add verilog defines to sv2v calls

2756a31

kamilrakoczy closed this Dec 21, 2021

Add UHDM frontend support for Yosys using Surelog #202

Add UHDM frontend support for Yosys using Surelog #202

Conversation

kamilrakoczy commented Dec 11, 2020

kgugala commented Dec 11, 2020

imphil commented Dec 18, 2020

kgugala commented Dec 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imphil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mithro commented Jan 10, 2021

tjurtsch commented May 14, 2021

olofk commented May 20, 2021

olofk commented May 21, 2021

olofk commented May 24, 2021

kgugala commented May 25, 2021

olofk commented May 25, 2021

tgorochowik commented May 28, 2021

olofk commented May 31, 2021

tgorochowik commented May 31, 2021

MikePopoloski commented May 31, 2021

olofk commented May 31, 2021

tgorochowik commented Jul 27, 2021

olofk commented Jul 28, 2021

olofk commented Dec 20, 2021

kamilrakoczy commented Dec 21, 2021