Support LitePCIe on Lattice ECP5 SERDES (with open toolchain)

[mithro]

Original message/request

I'm opening this bug to track the progress of this.

It is my understanding that;
(a) Project Trellis has all the documentation on the Lattice ECP5 SERDES.
(b) @whitequark has gotten the Lattice ECP5 SERDES to lock onto the PCIe bit stream.
(c) There is still a lot of work left to do.

---

Summary of Possible Funding and Work (last updated on 2025-01-09):

Total Bounty Pool:

• Current Offers: 11,400 EUR (400 + 1000 + 6000 + 2000 + 2000).
• Realistic Target: 8000 EUR (as suggested by @whitequark, but still considered low).

---

Summary Table

Funders:

Developer	Bounty Offer	Conditions/Requests
@snajpa	400 EUR (one-time?)	Requests a summary of current progress and known issues.
@motylewski	1000 EUR (upon completion by 2025)	Requires completion by 2025-06-30.
@madscientist159	6000 EUR (conditional)	Requires core to work on Arctic Tern and POWER9 systems, with MMIO support.
@teknoman117	2000 EUR	Core must be integrated with LitePCIe and demonstrated with MMAP on an ECP5 board.
@enjoy-digital	2000 EUR	Core must be integrated with LitePCIe and demonstrated with MMAP on an ECP5 board.

Developers:

Developer	Role	Conditions/Requests
@whitequark	Technical contributor	Willing to work but requires 8000 EUR (still considered low). Unlikely by H1 2025.

---

Detailed Requests

@motylewski:

• Request: Sponsors development of a free ECP5-PCIe core with a free Verilog toolchain for LFE5UM(G).
• Offer:
  • Now: One LFE5UM-45F-VERSA-EVN(G) eval board (250-400 EUR) OR 200 EUR if no hardware is needed.
  • Upon Completion: 800 EUR if the deadline 2025-06-30 is met.
• Background: Needs an open-source PCIe core with a Wishbone interface by July 2025 to avoid purchasing Lattice's core (837 USD/year).

Raptor Engineering:

• Request: Offers 6000 EUR, conditional on:
  1. Core works on Arctic Tern platform (ECP5-85 + 1GB RAM).
  2. Functions as a PCIe slave on POWER9 systems (e.g., Blackbird).
  3. Synthesizable and routable on POWER hosts without binary blobs.
  4. Basic MMIO support (BAR mapping and Wishbone read/write).
• Additional Support: Provides Arctic Tern module, carrier, and Blackbird host for development.

---

Key Points

• Total Bounty Pool: 11,400 EUR (current offers), exceeding the realistic target of 8000 EUR.
• Work: @whitequark is willing to work but needs sufficient funding and is unlikely to complete by H1 2025.
• Hardware: @madscientist159 offers hardware support (Arctic Tern and Blackbird host).
• Deadline: 2025-06-30 for @moylewski's offer; no strict deadline for others.

[mithro]

@whitequark, @enjoy-digital & @daveshah1 - Any chance you could add a few more details here?

[whitequark]

I'll fill you in next time I work on Yumewatari.

[mithro]

Yumewatari can be found at https://github.com/whitequark/Yumewatari

[rowanG077]

I really want this! @whitequark is Yumewatari PCIe implementation complete? I really can't tell just from the code since I don't know migen.

[whitequark]

No. It is not even close to complete. There are weeks or more realistically months of work left.

[snajpa]

@whitequark I've looked and it seemed like some fun stuff, could you possibly please sum up in a few bulletpoints to where you've gotten, what sources did you use and what the known bugs/design issues are?

I mean, I will eventually figure that out from the src itself, but doing this would save me (and others) a ton of time, I'd hugely appreciate if you could, please ;)

[teknoman117]

Are there any updates on this front? I'd love to try to help. I should be getting the ECP5-5G Versa sometime in January (8 week lead time is no fun), however I don't have access to one of those fancy keysight scopes. It would be nice to have something with PCIe support on the fully open toolchains.

[enjoy-digital]

@teknoman117: you can have a look and maybe contribute to the ECP5-PCIe project that is trying to create an open-source PCIe PHY on ECP5: https://github.com/ECP5-PCIe/ECP5-PCIe

[enjoy-digital]

Closing since not realistic for now, we could re-discuss this when ECP5-PCIe project will be functional/validated or another project will be available.

[madscientist159]

@teknoman117 We're also looking for this functionality, we basically just need a PCIe slave that we can wire up to e.g. Wishbone for MMIO. It looks like the ECP5-PCIe project has things to the point where the link is training and staying online, but I'm not sure how to contact the author(s) directly.

My suspicion is this is something we could all collaborate on to complete in a relatively short period of time, but we need communication open first. 😄

[cz172638]

@teknoman117 We're also looking for this functionality, we basically just need a PCIe slave that we can wire up to e.g. Wishbone for MMIO. It looks like the ECP5-PCIe project has things to the point where the link is training and staying online, but I'm not sure how to contact the author(s) directly.

My suspicion is this is something we could all collaborate on to complete in a relatively short period of time, but we need communication open first.

i pinged him via email (git show) today, thanked him for his work and asked for news.

[teknoman117]

I also did the same today, I wonder if I'll hear back. I haven't dug in too deeply, but it looks like it's getting there. Some of the tests are about processing DLL packets. The next layer up is the TLP layer, which LitePCIe already implements. I also wanted to know how to get the adapter board in their repository built. I ordered one the EVN boards for the ECP5-5G, looks like the Versa 5G was discontinued when the Certus-NX boards started coming out.

…

On Mon, Jan 18, 2021 at 5:07 AM Jiri Kastner ***@***.***> wrote: @teknoman117 <https://github.com/teknoman117> We're also looking for this functionality, we basically just need a PCIe slave that we can wire up to e.g. Wishbone for MMIO. It looks like the ECP5-PCIe project has things to the point where the link is training and staying online, but I'm not sure how to contact the author(s) directly. My suspicion is this is something we could all collaborate on to complete in a relatively short period of time, but we need communication open first. i pinged him via email (git show) today, thanked him for his work and asked for news. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACPTDE5NDGKAJPQCCK6GSTS2QXA7ANCNFSM4HNQ7OYA> .

[motylewski]

I would like to sponsor further development of a free ECP5-PCIe core with free Verilog toolchain for LFE5UM(G) (ECP5UM5 or ECP5UMG) with:

• now - one LFE5UM-45F-VERSA-EVN(G) eval board (costs 250-400 EUR) OR just 200 EUR if lead developer does not need such hardware
• upon completion - 800 EUR if the deadline 2025-08-10 is kept.

In case someone is seriously interested in participating, please get in touch with me till 15.01.2025 so I may add additional LFE5UM evaluation board to my planned order.

Background:

I have already ported my old ISA-interfaced FPGA SW to PCIEx1 using Lattice LFE5UM-45F-VERSA-EVN evaluation board.
I am using their core in evaluation mode (works for 4 hours, then locks), together with their demo sources - Verilog wrapper providing Wishbone4 interface.
My Verilog code is connected as Wishbone slave, 16bit, 125 MHz, no DMA. I am using the Lattice Diamond 1 year license which came with LFE5UM-45F-VERSA-EVN.
The wrapper code is available upon registration: https://www.latticesemi.com/view_document?document_id=51397

Now, just before I deliver the first real prototype (August 2025) I will need either to buy the Lattice PCIE core (it costs 837 USD per year) or have open source PCIE core with simple WB interface ready.

Tomasz Motylewski IT-Services, Poland
[email protected]

[whitequark]

800 EUR is an unrealistic offer for a project of this size and complexity. I have both the skills required and an interest in building this core, but I cannot justify it for less than 10 times that amount on the grounds of cost of living alone.

[snajpa]

I'd really love to be able to pitch in more but I can't reliably promise more than 400 EUR (every 6 months until the project hits general usability). That much I can commit to right now. Perhaps there are more people who could help? I'd hope?

[teknoman117]

@whitequark Is 8000 EUR actually what you need to build the core? I can't solo fund it, but you can put me down for 2000 EUR of that if others join.

[whitequark]

I would have to scope out the details before committing. LUNA (written in Amaranth) already has a working PIPE implementation, which is a really useful component comprising a significant part of the PHY. (LUNA's is for USB, but PCIe is very similar.)

Given that LitePCIe implements TLP handling and LUNA implements much of the lower PHY as well as macro instantiation, one could expect that implementing DLLP handling and PCIe-specific parts of lower PHY (and debugging it all, of course) would be enough.

This is still a very complex and ambitious project--I would expect that you would be able to collect everyone who's ever implemented PCIe upper PHY in a single reasonably large room--and it's probably worth a lot more than that, but since I'm personally interested in writing a PCIe core I'm willing to compromise here.

(There is no way I'll be able to make the H1 2025 deadline due to existing commitments.)

[madscientist159]

@whitequark Raptor is willing to make up the remaining 6000 EUR differerential, as long as the following conditions can be met by end of year 2025:

• The resulting core needs to work on our Arctic Tern platform [1] (ECP5-85 + 1GB RAM. so plenty of space)
• It also needs to work as a PCIe slave on at least one of our current POWER9 product lines, e.g. Blackbird [2] -- this is important as the POWER PHB is very strict on its adherence to the PCIe specification, and will generate EEH events if the specification is violated. Since the intent is to use the Arctic Tern with POWER9 hosts, it is important we don't immediately cause a fault on the controller end of things 😉
• It needs to be synthesizeable and routable on a POWER host, preferably as part of our existing Kestrel BMC solution [3] (based on an older version of LiteX that is updated as necessary) with no binary blobs needed to use the core
• Finally, all we really need right now is basic MMIO support. If we can map a BAR on the host and read/write (DMA) to slave memory via Wishbone that would be perfect

We are also willing to provide an Arctic Tern module, carrier, and a Blackbird host to support the work, which would be yours to keep if desired after the work is complete.

Does that seem reasonable?

[1] https://www.raptorcs.com/content/AT1PC2/intro.html
[2] https://www.raptorcs.com/content/BK1B02/intro.html
[3] https://gitlab.raptorengineering.com/kestrel-collaboration

[enjoy-digital]

Nice to see interest in funding and development!

Enjoy-Digital is also willing to contribute 2000 EUR, with the only condition that the PHY is compatible with LitePCIe and demonstrated on an ECP5 board. This should already be covered by other requests and aligns with my understanding of @whitequark’s proposed development.

I’ve updated the original request from @mithro with a summary of the current funding and development proposals to help track things. This will also make it easier to collect additional funding, as I believe @whitequark’s request is still on the low side to complete this work effectively.

[motylewski]

Hi @whitequark ,
I am fully aware that my meager 800 EUR+board was just an incentive, bait for someone who wanted to do it anyway as a hobby project, to start it now. I had thought about starting some share-funding - just to see it already happened :-)
Amazing.

I do see there exist many pieces of that puzzle (for example https://codeberg.org/ECP5-PCIe/ECP5-PCIe) and you surely are one of the very few who can put the puzzle together.

My perspective: I need basic MMIO in one BAR mimimum 256 KB address space. PCIe x1. 16 bit Wishbone is nice to have. There is no need to pipeline more than 64 bit (so when a host issues 64 bit read or write it gets translated to 4 x 16 bit WB transactions). I am happy with Wishbone classic ACK ON/OFF 50% duty cycle or lower speed. I will be delaying WB ACK upon some RD or WR cycles up to 2-3 microseconds (external bus IO).

I need that till early August 2025. If not ready, no problem, I buy 1 year license from Lattice for 800-900 EUR and use it.

By end of February I need to know whether I may use ECP5UM-45F or I do you target ECP5UM5G-45F?

Bare minimum would be implementing only subset of what Lattice IP core currently does:

• In transmit, user creates TLPs without ECRC, LCRC, or sequence number
• In receive, user receives valid TLPs without ECRC, LCRC, or sequence number
• Credit interface for transmit and receive for PH, PD, NPH, NPD, CPLH, CPLD credit types
• Upstream/downstream, single function endpoint topology
• Higher layer control of LTSSM through ports
• Access to select configuration space information through ports

And I would like to know ASAP (this week) if I need to order an LFE5UM-45F-VERSA-EVN for you?
I am ordering one for me from Newark USA anyway (discounted 232 USD), so I could just order 2.

[eibach]

I would offer my help on this project. I could only work on this as a side/hobby project, I have no experience implementing stuff on the PCIe phy level but did a lot of stuff higher up. I have a little LiteX experience, know some Verilog, also lots of u-boot and linux kernel driver experience. So if I could help here, I would be happily on board. I have no real interest in the money but I would need a Vera board as my ECP5 platform is without Serdes. So if I can do some tasks in addtion to @whitequark's work. I would love to be part of the team.

[teknoman117]

@enjoy-digital saw you updated the issue. My only conditions are the same as yours. Works with LitePCIe and is demonstrated on an ECP5 / ECP5-5G board.

[whitequark]

That is a lot of interest in this issue! I'll have to spend some time putting together a detailed investigation of the current state and a work plan now that there are specific commitments to realistic amounts of funding. I am fairly busy with existing commitments right now so it might take a bit, but this issue will definitely remain on my radar as it's something that is highly aligned with my goals.

Is anyone here able to provide a PCIe analyzer? Nobody here is asking for Gen2 or multiple lanes so even the cheapest eBay option will work. I know that a USB 3 analyzer has been invaluable for the LUNA work, and I also remember that during the development of Yumewatari, lacking one was a major time sink.

[eibach]

As soon as I have a versa board, I could do some tests how far LUNA and ECP5-PCIe take us. An analyzer would certainly be invaluable.

[whitequark]

I have a Versa ECP5-5G that I could probably send over if you promise me to return it whenever you're not using it. I'd have to make sure I have some other board to run stuff on in the meantime but I can probably find one somewhere...

[eibach]

Thanks Catherine, that's awesome. As Tomasz offered to sponsor one and I guess we will need at least two in the long run anyway: @motylewski what do you think?

[teknoman117]

Nobody here is asking for Gen2 or multiple lanes

@whitequark gen.2 and multiple lanes would be nice :) but I do recognize the significant increase in complexity for that and I don't consider it a requirement of my offer. I unfortunately don't have access to a pcie analyzer.

I am willing to provide an ECP5-5G EVN board + the multi-lane adapter board (or equivalent) from ECP5-PCIe to anyone who can materially contribute to multi lane support (capped at like 3).

I would need some lead time so I can understand the design and place orders for parts.

@madscientist159 Is the dedicated external PLL chip an absolute necessity? I noticed in the Arctic Tern schematic that the pcie refclk is attached directly to the ECP5. Both the versa and the ECP5-PCIe repos's adapter use an external PLL.

[whitequark]

Getting a Versa is likely a better choice.

It has a PLL to double the incoming PCIe clock, since it has better jitter performance than the ones in the ECP5.

In particular I want to underscore that the jitter performance of ECP5 PLLs is perfectly satisfactory for either USB3 and PCIe and this sentence is based on someone's misunderstanding of the protocol(s). On Versa you can ignore the external PLL, it's not necessary and really just adds complexity.

[teknoman117]

I recognize the versa is a better choice in general but it does only have one PCIe lane 😛.

[whitequark]

Given the options of going x1 to x2 and Gen1 to Gen2, I would pick the latter any time of day.

[eibach]

I would also prefer Versa, as it has been designed for this task. We can switch to another platform when the basics are working.

[eibach]

My PCIe setups have a tendency to escalate by themselves 😆
Sadly the loaned 8k€ Microsemi PCIe switch eval board died on my bench, so I am somewhat reluctant to accept loaned boards 😟

[motylewski]

Hi @eibach ,

Since @whitequark has Versa ECP5-5G I will send you the ECP5UM VERSA I am ordering just now.
So both of you have at least one board.

[eibach]

Awesome! So I can do some research to prepare before Catherine starts.

[eibach]

By the way what happened to this project: https://github.com/enjoy-digital/pcie_analyzer

[eibach]

@whitequark since parts of the sponsorship are bound to this: I would base my test setup on migen/LiteX, right?

[whitequark]

I would base my test setup on migen/LiteX, right?

I will expect there to be a significant Amaranth component in order to reuse code from LUNA, and naturally once that is in place I will continue to add new code mostly in Amaranth. Of course the end result would be usable from LiteX.

[eibach]

This Github issue is probably not the right place for further technical communication. @whitequark what is your medium of choice? Discord?

[whitequark]

Matrix or IRC

[eibach]

I have created a matrix space where I will document my results. Everybody is welcome to continue discussing the development there: https://matrix.to/#/#open-pcie-core:matrix.org

[Johnsel]

Offered time in exchange for hw via email but not sure if it arrived, didn’t get an answer