Support gziped Liberty (.lib) files

[hpretl]

Description

Liberty files are huge, which is not ideal when multiple PDKs are installed on a drive. If OL2 supported zipped .lib files naturally, that would be a big thing and super helpful. Plus, should not be super difficult, as OpenROAD seems to support zipped Liberty files already.

Proposal

Either unzip needed file in /tmp on request, or just make the read routines zip-capable.

[hpretl]

See also iic-jku/IIC-OSIC-TOOLS#88

[hpretl]

@donn Here a small wish for 2025. Happy new year!

[donn]

Happy New Year to you too-- where exactly though because I know you can just put in .lib.gz in most places and the flow would work

[hpretl]

Here's the result of a quick test. gzip'ed all .lib, modified the config.tcl accordingly, and ran the smoke-test:

/foss/designs > openlane --smoke-test
[13:45:49] INFO     Starting a new run of the 'Classic' flow with the tag 'RUN_2025-01-01_13-45-49'.                                                                    flow.py:628
[13:45:49] INFO     Starting…                                                                                                                                     sequential.py:278
───────────────────────────────────────────────────────────────────────────────── Verilator Lint ──────────────────────────────────────────────────────────────────────────────────
[13:45:49] VERBOSE  Running 'Verilator.Lint' at '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/01-verilator-lint'…                     step.py:1122
[13:45:50] VERBOSE  Logging subprocess to '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/01-verilator-lint/verilator-lint.log'…        step.py:1318
- V e r i l a t i o n   R e p o r t: Verilator 5.030 2024-10-27 rev v5.030
- Verilator: Built from 0.368 MB sources in 444 modules, into 0.099 MB in 5 C++ files needing 0.000 MB
- Verilator: Walltime 0.032 s (elab=0.005, cvt=0.007, bld=0.000); cpu 0.022 s on 1 threads; alloced 31.770 MB
─────────────────────────────────────────────────────────────────────────── Lint Timing Errors Checker ────────────────────────────────────────────────────────────────────────────
[13:45:50] VERBOSE  Running 'Checker.LintTimingConstructs' at                                                                                                          step.py:1122
                    '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/02-checker-linttimingconstructs'…
[13:45:50] INFO     Check for Lint Timing Errors clear.                                                                                                              checker.py:404
─────────────────────────────────────────────────────────────────────────────── Lint Errors Checker ───────────────────────────────────────────────────────────────────────────────
[13:45:50] VERBOSE  Running 'Checker.LintErrors' at '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/03-checker-linterrors'…             step.py:1122
[13:45:50] INFO     Check for Lint errors clear.                                                                                                                     checker.py:132
────────────────────────────────────────────────────────────────────────────── Lint Warnings Checker ──────────────────────────────────────────────────────────────────────────────
[13:45:50] VERBOSE  Running 'Checker.LintWarnings' at '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/04-checker-lintwarnings'…         step.py:1122
[13:45:50] INFO     Check for Lint warnings clear.                                                                                                                   checker.py:132
────────────────────────────────────────────────────────────────────────────── Generate JSON Header ───────────────────────────────────────────────────────────────────────────────
[13:45:50] VERBOSE  Running 'Yosys.JsonHeader' at '../../tmp/tmpwltsza4eopenlane2/smoke_test_design/runs/RUN_2025-01-01_13-45-49/05-yosys-jsonheader'…                 step.py:1122
Classic - Stage 5 - Generate JSON Header ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/78 0:00:00
Traceback (most recent call last):
  File "/usr/local/bin/openlane", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openlane/flows/cli.py", line 442, in pdk_resolve_wrapper
    return f(*args, pdk_root=pdk_root, pdk=pdk, scl=scl, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openlane/__main__.py", line 437, in cli
    run_included_example(ctx, smoke_test, example, **run_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openlane/__main__.py", line 266, in run_included_example
    run(
  File "/usr/local/lib/python3.10/dist-packages/openlane/__main__.py", line 147, in run
    state_out = flow.start(
  File "/usr/local/lib/python3.10/dist-packages/openlane/flows/flow.py", line 657, in start
    final_state, step_objects = self.run(
  File "/usr/local/lib/python3.10/dist-packages/openlane/flows/sequential.py", line 322, in run
    current_state = step.start(
  File "/usr/local/lib/python3.10/dist-packages/openlane/steps/step.py", line 1150, in start
    views_updates, metrics_updates = self.run(state_in_result, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openlane/steps/pyosys.py", line 320, in run
    views_updates, metrics_updates = super().run(state_in, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openlane/steps/pyosys.py", line 222, in run
    cmd = self.get_command(state_in)
  File "/usr/local/lib/python3.10/dist-packages/openlane/steps/pyosys.py", line 313, in get_command
    return super().get_command(state_in) + ["--output", out_file]
  File "/usr/local/lib/python3.10/dist-packages/openlane/steps/pyosys.py", line 281, in get_command
    libs_synth = self.toolbox.remove_cells_from_lib(
  File "/usr/local/lib/python3.10/dist-packages/openlane/common/toolbox.py", line 399, in remove_cells_from_lib
    for line in input_lib_stream:
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

[hpretl]

One step further:

Changing this in toolbox.py looks easy enough:

input_lib_stream = gzip.open(file, mode="rt")

But then Yosys is tripping:

[14:01:35] VERBOSE  Running 'Yosys.Synthesis' at '../../tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis'…                   step.py:1122
[14:01:35] VERBOSE  Logging subprocess to '../../tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis/yosys-synthesis.log'…      step.py:1318
[ERROR] Black-box model '/foss/pdks/sky130A/libs.ref/sky130_fd_sc_hd/lib/sky130_fd_sc_hd__tt_025C_1v80.lib.gz' has an unrecognized file extension.

/----------------------------------------------------------------------------\
|  yosys -- Yosys Open SYnthesis Suite                                       |
|  Copyright (C) 2012 - 2024  Claire Xenia Wolf <[email protected]>         |
|  Distributed under an ISC-like license, type "license" to see terms        |
\----------------------------------------------------------------------------/
Yosys 0.48 (git sha1 aaa534749, g++ 11.4.0-1ubuntu1~22.04 -fPIC -O3)
[14:01:36] ERROR    Subprocess had a non-zero exit.                                                                                                                    step.py:1364
[14:01:36] ERROR    Last 8 line(s):                                                                                                                                    step.py:1369
                    [ERROR] Black-box model '/foss/pdks/sky130A/libs.ref/sky130_fd_sc_hd/lib/sky130_fd_sc_hd__tt_025C_1v80.lib.gz' has an unrecognized file extension.

                     /----------------------------------------------------------------------------\
                     |  yosys -- Yosys Open SYnthesis Suite                                       |
                     |  Copyright (C) 2012 - 2024  Claire Xenia Wolf <[email protected]>         |
                     |  Distributed under an ISC-like license, type "license" to see terms        |
                     \----------------------------------------------------------------------------/
                     Yosys 0.48 (git sha1 aaa534749, g++ 11.4.0-1ubuntu1~22.04 -fPIC -O3)

[14:01:36] ERROR    Full log file: '../../tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis/yosys-synthesis.log'              step.py:1372
Classic - Stage 6 - Synthesis ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  5/78 0:00:00
[14:01:36] ERROR    The following error was encountered while running the flow:                                                                                     __main__.py:163
                    Synthesis: subprocess (255, ['yosys', '-y', '/usr/local/lib/python3.10/dist-packages/openlane/scripts/pyosys/synthesize.py', '--',
                    '--config-in', '/tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis/config.json', '--extra-in',
                    '/tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis/extra.json', '--output',
                    '/tmp/tmp3cq1yop4openlane2/smoke_test_design/runs/RUN_2025-01-01_14-01-35/06-yosys-synthesis/spm.nl.v']) failed
[14:01:36] ERROR    OpenLane will now quit.

[hpretl]

It looks like Yosys is already supporting gzip'ed Verilog files, so I opened a ticket to support zipped Liberty files as well.

YosysHQ/yosys#4830

[donn]

I will treat this is a bug so I can push it as 2.3.2. In general we do want to support gzipped files wherever possible.

While I'm there, I'll look into your httpx issue as well.

[hpretl]

Super, thank you!

[donn]

So as it turns out…

With the Linux Standard Base gzopen, if you pass an uncompressed file for reading, it still opens it, no biggie.

Python's gzip.open, however, will fail upon the first read.

It's always something with this language…

[hpretl]

We are now one step further, but the flow fails when running the first STA. OpenSTA should support .lib.gz (see The-OpenROAD-Project/OpenROAD#6484 (comment)) so I don't know what is going on. Yosys seems fine BTW. Here the fail log:

Starting…
Running 'Verilator.Lint' at 'runs/RUN_2025-01-07_17-11-37/01-verilator-lint'…
Logging subprocess to [repr.filename]'runs/RUN_2025-01-07_17-11-37/01-verilator-lint/verilator-lint.log'[/repr.filename]…
Running 'Checker.LintTimingConstructs' at 'runs/RUN_2025-01-07_17-11-37/02-checker-linttimingconstructs'…
Check for Lint Timing Errors clear.
Running 'Checker.LintErrors' at 'runs/RUN_2025-01-07_17-11-37/03-checker-linterrors'…
Check for Lint errors clear.
Running 'Checker.LintWarnings' at 'runs/RUN_2025-01-07_17-11-37/04-checker-lintwarnings'…
Check for Lint warnings clear.
Running 'Yosys.JsonHeader' at 'runs/RUN_2025-01-07_17-11-37/05-yosys-jsonheader'…
Logging subprocess to [repr.filename]'runs/RUN_2025-01-07_17-11-37/05-yosys-jsonheader/yosys-jsonheader.log'[/repr.filename]…
Running 'Yosys.Synthesis' at 'runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis'…
Logging subprocess to [repr.filename]'runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/yosys-synthesis.log'[/repr.filename]…
Parsing synthesis checks…
Running 'Checker.YosysUnmappedCells' at 'runs/RUN_2025-01-07_17-11-37/07-checker-yosysunmappedcells'…
Check for Unmapped Yosys instances clear.
Running 'Checker.YosysSynthChecks' at 'runs/RUN_2025-01-07_17-11-37/08-checker-yosyssynthchecks'…
Check for Yosys check errors clear.
Running 'Checker.NetlistAssignStatements' at 'runs/RUN_2025-01-07_17-11-37/09-checker-netlistassignstatements'…
Running 'OpenROAD.CheckSDCFiles' at 'runs/RUN_2025-01-07_17-11-37/10-openroad-checksdcfiles'…
'PNR_SDC_FILE' is not defined. Using generic fallback SDC for OpenROAD PnR steps.
'SIGNOFF_SDC_FILE' is not defined. Using generic fallback SDC for OpenROAD PnR steps.
Running 'OpenROAD.CheckMacroInstances' at 'runs/RUN_2025-01-07_17-11-37/11-openroad-checkmacroinstances'…
No macros found, skipping instance check…
Running 'OpenROAD.STAPrePNR' at 'runs/RUN_2025-01-07_17-11-37/12-openroad-staprepnr'…
No SCL lib files found for nom_tt_025C_1v80.
Starting STA for the nom_tt_025C_1v80 timing corner…
No SCL lib files found for nom_ss_100C_1v60.
Skipping corner nom_ss_100C_1v60 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for nom_ff_n40C_1v95.
Skipping corner nom_ff_n40C_1v95 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for min_tt_025C_1v80.
Skipping corner min_tt_025C_1v80 for STA (identical to nom_tt_025C_1v80 at this stage)…
Logging subprocess to [repr.filename]'runs/RUN_2025-01-07_17-11-37/12-openroad-staprepnr/nom_tt_025C_1v80/sta.log'[/repr.filename]…
No SCL lib files found for min_ss_100C_1v60.
Skipping corner min_ss_100C_1v60 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for min_ff_n40C_1v95.
Skipping corner min_ff_n40C_1v95 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for max_tt_025C_1v80.
Skipping corner max_tt_025C_1v80 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for max_ss_100C_1v60.
Skipping corner max_ss_100C_1v60 for STA (identical to nom_tt_025C_1v80 at this stage)…
No SCL lib files found for max_ff_n40C_1v95.
Skipping corner max_ff_n40C_1v95 for STA (identical to nom_tt_025C_1v80 at this stage)…
Subprocess had a non-zero exit.
Last 10 line(s):
Warning: /foss/designs/example/dig/runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/counter.nl.v line 256, module sky130_fd_sc_hd__nor2_2 not found. Creating black box for _131_.
Warning: /foss/designs/example/dig/runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/counter.nl.v line 281, module sky130_fd_sc_hd__a21oi_2 not found. Creating black box for _136_.
Warning: /foss/designs/example/dig/runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/counter.nl.v line 287, module sky130_fd_sc_hd__and3_2 not found. Creating black box for _137_.
Warning: /foss/designs/example/dig/runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/counter.nl.v line 344, module sky130_fd_sc_hd__a31oi_2 not found. Creating black box for _147_.
Warning: /foss/designs/example/dig/runs/RUN_2025-01-07_17-11-37/06-yosys-synthesis/counter.nl.v line 668, module sky130_fd_sc_hd__dfrtp_2 not found. Creating black box for _212_.
Reading design constraints file at '/usr/local/lib/python3.12/dist-packages/openlane/scripts/base.sdc'…
[INFO] Using clock i_clk…
[INFO] Setting output delay to: 4
[INFO] Setting input delay to: 4
Error: base.sdc line 54, 'sky130_fd_sc_hd__inv_2' not found.

Full log file: 'runs/RUN_2025-01-07_17-11-37/12-openroad-staprepnr/nom_tt_025C_1v80/sta.log'
Failed STA for the nom_tt_025C_1v80 timing corner

[hpretl]

@donn Maybe you know where the fail happens, whether OL2 or OpenSTA related. If OpenSTA, then I can open a ticket there, but in the moment I don't know what is going on.