Print failure RAMBO firmware related?

[WarMachine]

My question may sound a bit out there, but is there a ceiling on the number of lines of code the firmware can handle?
My question stems from a consistent, repeatable, and predictable failure of my prints that have gcode with >225,000 lines.

A bit of backstory: Repetier Host v.90c/slicer 0.9.9/.80 firmware
I had a fairly simple part that I was able to print off successfully @.2mm layer height in pla (approx. 200,000 lines of code including start and end code), only to find out that pla would not meet the temperature requirements for the parts application, so I switched to abs. I made appropriate adjustments due to material differences and had no issues aside from what seemed to be a bit of lagging near the end of the print, the buffer steadily dropped from 16 with the passing of each layer as it got closer to the end. I was not overly pleased with the finish of the part. I decided to give the same part a slice @.1mm layer height and see how the end product came out, and that is when the trouble started.

The new code contains approx 500,000 lines (831 layers, 13 hour print time), and works great up until approx. the 225,000th line. I have checked the model for errors, resliced multiple times, reinstalled software/firmware (configuration.h settings corrected for my machine) from the ground up, and even tried different parts with roughly equivalent lines of code. I have attempted slowing down the print speed, and likewise speeding it up. Nothing seems to make it past approx. the 200,000 mark without the buffer dropping off steadily and the print eventually coming to a stop.

Prior to the print stopping, the buffered moves bounces from 16-14 as it seems to normally. Nearing the 200,000th line the buffer starts dropping into the 7-4 range for a few layers, then down to 2-1...lagging obviously. Eventually the printer stops, with the extruder still heated, fans still running, bed still heated...repetier host shows no communication errors, lcd displaying printing (or printing..XX% if from sd card). I have checked and swapped usb cables with no improvement, and even eliminated the computer all together and tried prints from the sd card. Each effort (regardless of the part or method of data delivery) die at or near the same point. The printer must be shut down and restarted in order to regain manual control.

Dry runs with the code sections work smoothly, as does printing sections of the code in the trouble area. I have not tried a dry run of the entire code.

I'm at a loss. I've burned my way through a couple of spools of abs and many, many hours of worthless printing time now trying to figure this out.
From my attempt at a process of elimination: I feel I have ruled out the computer, the host, the connection from computer to printer, the part, and the code. Leaving only the firmware. Am I overlooking something? Or have I just exceeded the ability of this setup? Any suggestions are welcome!

[WarMachine]

Also worth mentioning that I have tried changing settings in the configuration.h #define MOVE_CACHE_SIZE, #define MOVE_CACHE_LOW, and #define LOW_TICKS_PER_MOVE definitions. No changes I made had positive results.

FWIW: a dry run of the entire code fails at roughly the same point as the real deal.

[WarMachine]

Is there a different area I should have posted this?

[Eaglezsoar]

Your message has 38 views but no responses. I don't thing you are being ignored, it's just that no one has run into the problem you are having.
To get more views and a possible response, create a new thread under troubleshooting. I think it will get more exposure there and a possible
answer.

[JohnStack]

My response is no help...

Don't repost...it's ok. This is definitely an Ultimaker support post.......if you don't violate your customer's privacy, I would send them the STL.

Definitely an edge case but heh, edge cases make technology!

If you think of all the code and start eliminating things that are not dynamic, I think you would isolate everything down to serial coms and head movement. My WAG is something's getting left over in head movement or the geometry of it.

This one is going to be a b* to find. Sorry.

[WarMachine]

Your message has 38 views but no responses. I don't thing you are being ignored, it's just that no one has run into the problem you are having.
To get more views and a possible response, create a new thread under troubleshooting. I think it will get more exposure there and a possible
answer.

I definitely didn't mean to imply I felt as though I was being ignored. Perhaps just that I chose incorrectly where to post this.

My response is no help...

Don't repost...it's ok. This is definitely an Ultimaker support post.......if you don't violate your customer's privacy, I would send them the STL.

Definitely an edge case but heh, edge cases make technology!

If you think of all the code and start eliminating things that are not dynamic, I think you would isolate everything down to serial coms and head movement. My WAG is something's getting left over in head movement or the geometry of it.

This one is going to be a b* to find. Sorry.

I personally didn't want to repost this, this posting alone hogs enough space as it is, I tend to be overly wordy at times. Unfortunately, I am bound by a non disclosure agreement with this customer in particular, but am trying to seek guidance without being so vague as to shoot myself in the foot. The kicker for me is that on different versions of the part (and different parts entirely) that contain approx. the same line count....they all fail at or about the same point. Dry run or printing, sd card or usb...no errors to point me in any direction, just like somebody hits pause and throws away the remote.

Additionally unfortunate for me is that, as of typing this, I have lost the rights to this one part already with this customer. That part of this project is done but the problem still remains, and could limit future projects I attempt.

[JohnStack]

I would contact Ultimaker. Perhaps there is a mod you cam make to the DWG to get the point across.

Sorry you lost part of the project.

[WarMachine]

I would contact Ultimaker. Perhaps there is a mod you cam make to the DWG to get the point across.

Sorry you lost part of the project.

Did you mean UltiMachine?

[geneb]

I'm confused. Why would you suggest he contact the hardware vendor for a software issue?

If I was having this problem, the first thing I'd do is contact Repetier directly and explain to him what the problem was and see if he had any suggestions. He wrote the firmware, so he's going to know the most about it.

If that didn't get the issue solved, I'd try the older Marlin firmware to see if that resolved it.

Either way, good luck WM!

g.

[WarMachine]

I'm confused. Why would you suggest he contact the hardware vendor for a software issue?

If I was having this problem, the first thing I'd do is contact Repetier directly and explain to him what the problem was and see if he had any suggestions. He wrote the firmware, so he's going to know the most about it.

If that didn't get the issue solved, I'd try the older Marlin firmware to see if that resolved it.

Either way, good luck WM!

g.

My thoughts were to try and see if anyone else had encountered this before I ran it up the chain to Repetier. Started by trying to solve the problem myself with what I know, then seek outside input from folks that are more fluent than I am and actually have/use these machines aswell, and lastly, ring the bell at the source. It's not Repetiers fault that the parts I am trying to print have over half a million lines in the gcode, so I figured start small...but perhaps I should start typing up an email to send off to the address listed on Repetiers website.

For the most part, I am pleased with the Repeiter firmware, seems to function for me better than the Marlin firmware on other prints to date. I have already lost the part to someone with a much more expensive setup (unknown make/model), and at last report, they are having no issues with my design or code. I was told they imported my slicer gcode (minus my start-stop commands) and ran off 6 consecutive parts without a hitch. So the sense of urgency is gone, but the big question mark still remains for me.

Anyone know of a faster way to have the board simulate running the code without performing a real time "dry run"? I'd really like to see the results without having to wait 6-7 hours for it to fail.

[foshon]

Do you think the issue is buffer related, like the amount of lines already sent? If not, you could start the dry run a few thousand lines before the problem area. I did find a post on the reprap forums that speaks of related issues:

http://forums.reprap.org/read.php?146,67665,110979

If it were me, I would try to slice it with KISSlicer or Cura and see if anything changes before getting to crazy into it. Slic3r just absolutely sucks at some stuff, thankfully not much but still.

[WarMachine]

Do you think the issue is buffer related, like the amount of lines already sent? If not, you could start the dry run a few thousand lines before the problem area. I did find a post on the reprap forums that speaks of related issues:

http://forums.reprap.org/read.php?146,67665,110979

If it were me, I would try to slice it with KISSlicer or Cura and see if anything changes before getting to crazy into it. Slic3r just absolutely sucks at some stuff, thankfully not much but still.

Prior to the print stopping, the buffered moves bounces from 16-14 as it seems to normally. Nearing the 200,000th line the buffer starts dropping into the 7-4 range for a few layers, then down to 2-1...lagging obviously. Eventually the printer stops, with the extruder still heated, fans still running, bed still heated...repetier host shows no communication errors, lcd displaying printing (or printing..XX% if from sd card). I have checked and swapped usb cables with no improvement, and even eliminated the computer all together and tried prints from the sd card. Each effort (regardless of the part or method of data delivery) die at or near the same point. The printer must be shut down and restarted in order to regain manual control.

Dry runs with the code sections work smoothly, as does printing sections of the code in the trouble area.

I did run across that same topic on reprap while performing my standard consultation of the all knowing google. That is what started me into trying to make adjustments in the firmware settings, eliminating communication possibilities, and sectioning out code. Basically what firmed up my opinion that this is a hardware/firmware issue. I went as far as putting a fan in the base to make sure I wasn't overheating the board (had to remove the fan later, couldn't maintain heated bed temps). The memory leak reference really seems plausible. It happens when you try to make a one size fits all firmware, and I applaud the folks out there doing it, it takes far more forethought and knowledge than I am capable of.

As far as slicer goes...I have run across some cases where it makes some boneheaded mistakes...but the fact that the "other guy" was able to take my slicer derived gcode and make half a dozen good prints where I couldn't complete one steers me away from it being a slicer issue.

[foshon]

Unless you are using the exact same settings with the exact same version, I wouldn't steer away to quickly. Both of the other offerings are free, prolly worth a shot.

I'm a bit confused by the fan thing. The board is inside the base, but the fan cooled your hot end (was it a house fan?). I have a 50mm fan mounted over the hole under the lcd mount. It pulls air into the base and blows it over the RAMbo. That being said, I don't believe the issue is hardware related. Unless, it has something to do with noise because of position or a board component reacting to something.

You said you installed a fresh copy of the firmware, did you DL the latest or just reinstall what you already had?

[WarMachine]

Unless you are using the exact same settings with the exact same version, I wouldn't steer away to quickly. Both of the other offerings are free, prolly worth a shot.

I'm a bit confused by the fan thing. The board is inside the base, but the fan cooled your hot end (was it a house fan?). I have a 50mm fan mounted over the hole under the lcd mount. It pulls air into the base and blows it over the RAMbo. That being said, I don't believe the issue is hardware related. Unless, it has something to do with noise because of position or a board component reacting to something.

You said you installed a fresh copy of the firmware, did you DL the latest or just reinstall what you already had?

I am not totally understanding your first statement. Exact same settings and version for.....slicing? What am I to compare, one slicing software to another? I am willing to give one or more of the others a go, but will only try dry run printing the result, I'm not convinced enough that this is due to slicing software to invest another 1/4 spool of abs on it. I will do a dry run when I have a day or two of idle printing time available.

I'm not inclined to blame slicer due to the following: The gcode for the problem part was saved directly from Rep Host w/slicer (all updated to current release), was sent directly to someone else (minus my settings, start, and stop commands...just raw gcode), who had no issues printing off multiple parts without a hitch. Granted this is how the customer reported it to me, but being that we are still conducting on going business, I don't see any advantage for them to misreport the case. They are very pleased with the design, but we both agree it would have been easier for them to get the parts directly from me...the "other guy" is about 2x as much.

On the fan subject: I had placed a 40mm fan (externally powered and pwm controlled) inside the base behind the lcd (my board location) thinking I was possibly overheating the board (fan running= 25c inside the base with 80c bed temp). As I stated previously, I was unable to maintain heated bed temp and the print failed at the same point regardless, so I pulled the fan back out.

As for software/firmware: I had recently updated Rep Host and still had the installation files for the new and the past versions, thinking this could be a bug within the new version, I scrubbed it. Reinstalled the previous version. Resliced and tried again. Matched up delta settings and reloaded firmware. Tried again. Tried via sd card with another new slice. Downloaded a fresh copy of the firmware, matched up delta settings, and tried again. Tried old slicer settings. Scrubbed old Rep Host and installed new version. Tried again. Tried via sd card. Downloaded fresh Rep Host and tried again. I am sure there are a few downloads/reloads I skipped in my replay. Basically I tried every combination of software/firmware/communication method I could come up with. New slices every time with different infill density, speed, temps, retracts...all fail at or about the same line of code. I would say all of the failed parts sitting in front of me are within .2mm of the same point of failure.

And just as a reminder: I have tried other parts...drastically different size/shape/settings....roughly equal gcode line count, and had them fail likewise.

Is there anyone with a similar line count (>500,000) of code that can report success or failure?

[WarMachine]

Or for that matter, >225,000 lines? The part prints off @ .2mm layers (half the number of lines) but I experience drastic buffering/lag issues near the completion. These machines are capable of impressive physical dimensions of printing, but perhaps I have found the limit of what my machine can do.

[Polygonhell]

I've printed items with almost 2 million lines of GCode without issues.
A pretty good test to see if your having connection issues is to copy the file to SDCard from repetier host with all the logging enabled, it will take a LONG time, but if you see a significant number of red error messages in the log, you have a USB noise issue, and I'd start by changing the USB cable.
Your currently thinking that it can't be that because it stops near the same line every time, there was a bug in earlier repetiers where if the top bit of a command was corrupted, it would misidentify the packet as binary, three of us could reproduce it here on the same file at more or less the same line, but the issue was just noise, it can be surprisingly repeatable.

[WarMachine]

I've printed items with almost 2 million lines of GCode without issues.
A pretty good test to see if your having connection issues is to copy the file to SDCard from repetier host with all the logging enabled, it will take a LONG time, but if you see a significant number of red error messages in the log, you have a USB noise issue, and I'd start by changing the USB cable.
Your currently thinking that it can't be that because it stops near the same line every time, there was a bug in earlier repetiers where if the top bit of a command was corrupted, it would misidentify the packet as binary, three of us could reproduce it here on the same file at more or less the same line, but the issue was just noise, it can be surprisingly repeatable.

Prior to the print stopping, the buffered moves bounces from 16-14 as it seems to normally. Nearing the 200,000th line the buffer starts dropping into the 7-4 range for a few layers, then down to 2-1...lagging obviously. Eventually the printer stops, with the extruder still heated, fans still running, bed still heated...repetier host shows no communication errors, lcd displaying printing (or printing..XX% if from sd card). I have checked and swapped usb cables with no improvement, and even eliminated the computer all together and tried prints from the sd card. Each effort (regardless of the part or method of data delivery) die at or near the same point. The printer must be shut down and restarted in order to regain manual control.

As for software/firmware: I had recently updated Rep Host and still had the installation files for the new and the past versions, thinking this could be a bug within the new version, I scrubbed it. Reinstalled the previous version. Resliced and tried again. Matched up delta settings and reloaded firmware. Tried again. Tried via sd card with another new slice. Downloaded a fresh copy of the firmware, matched up delta settings, and tried again. Tried old slicer settings. Scrubbed old Rep Host and installed new version. Tried again. Tried via sd card. Downloaded fresh Rep Host and tried again. I am sure there are a few downloads/reloads I skipped in my replay. Basically I tried every combination of software/firmware/communication method I could come up with. New slices every time with different infill density, speed, temps, retracts...all fail at or about the same line of code. I would say all of the failed parts sitting in front of me are within .2mm of the same point of failure.

I absolutely DO NOT mean to come off as unappreciative of the input, but covered those points already. I know my word heavy posts lead to readers just skimming. Tried with new version of Rep Host/slicer, tried with old version. No errors are shown real time on screen (commands, info, warnings, errors, and ack all enabled) or in log. The log just stops at the failure point, like it is waiting for the go ahead to continue. Tried different usb cables, usb cable and sd card, sd card alone (no physical communication link to computer).

There is a part that I just finished the model for that slices out at a little less than 2 million lines @ .1mm...but I am not willing to try printing until I can break the glass ceiling of 225k.

Aren't you running your own flavor of firmware? Or am I mistaken.

[Polygonhell]

Yes I use the version of the firmware I branched off of the main repetier, but it's unlikely to be significantly different from the current SeeMeCNC version.

As I said there was an issue in the firmware that did something like this, looking at the log after the failure the computer would be always re attempting to send one line over and over.
There was a fix out in for it, but this maybe a similar issue.

If you look at the repetier code, you can think of it having 2 halfs, there is an input portion that deals with the byte stream from USB or SDCard, all of that is run even if you just use repetier host to copy a file to the SD card, this was where the fixed bug was. If you try and copy the large file to the SDCard using repetier host, not using the SDCard reader on your computer, does it complete. This will take probably 10's of minutes, but it's a useful datapoint.

If you can reliably do that, the odds are it's a software issue in the motion code, and you can probaly come up with a repro with less GCode. If you can isolate a small piece of GCode that is causing the issue, then it will make fixing the issue easier.

There isn't really anything that can leak in the repetier code, it doesn't allocate memory it just works with fixed buffers, so it isn't likely that the issue is some ceiling with number of lines.

What I would suggest is opening an issue on the primary repetier github, but to get it addressed, you'll need to provide him with enough information to identify the issue in the code, since it's unlikely he'll be able to run your 225K lines of GCode on an identically configured printer.

The queue draining is usually up indicative of a lot of small moves, and may just be an artifact of that part of the print.

[WarMachine]

As I said there was an issue in the firmware that did something like this, looking at the log after the failure the computer would be always re attempting to send one line over and over.
There was a fix out in for it, but this maybe a similar issue.

That would be one difference I note so far. My log files just stop cold. No errors or attempted resends. Just dead stop.

If you look at the repetier code, you can think of it having 2 halfs, there is an input portion that deals with the byte stream from USB or SDCard, all of that is run even if you just use repetier host to copy a file to the SD card, this was where the fixed bug was. If you try and copy the large file to the SDCard using repetier host, not using the SDCard reader on your computer, does it complete. This will take probably 10's of minutes, but it's a useful datapoint.

I tried saving to the sd card both ways...saving the gcode to a file and copying to the sd card locally on the computer, and loading it directly on to the sd card remotely via Rep Host with the card in the printer. Both attempts failed at the same point I've come to expect.

If you can reliably do that, the odds are it's a software issue in the motion code, and you can probaly come up with a repro with less GCode. If you can isolate a small piece of GCode that is causing the issue, then it will make fixing the issue easier.

Sectioning out the code 10mm prior to and after the usual failure point perform fine. Printing and dry run both. Am I mistaken in thinking that if it were a code issue, it would show itself? Also, the buffer doesn't drop out while performing a section print as it does on the complete code.

What I would suggest is opening an issue on the primary repetier github, but to get it addressed, you'll need to provide him with enough information to identify the issue in the code, since it's unlikely he'll be able to run your 225K lines of GCode on an identically configured printer.

At that rate I'm just SOL. Submitting the code is not an option, thus my attempt at detailed descriptions of what the result is, and the steps I've taken so far to try for a correction.

The queue draining is usually up indicative of a lot of small moves, and may just be an artifact of that part of the print.

Yet another difference I am seeing relates to this. When nearing the completion of the same part sliced at .2mm (once again, that same 200k ball park), the buffer begins to drop and lagging ensues. Physically on the part, this would be identical to the beginning of the print, and no buffering issues are present at the start. In fact, these are the longest moves of the entire print. When printing the .1mm sliced version the kill zone of the printed part is also in an area that has fairly long low count moves, but yet the buffer drops down, and eventually stops. If I were to compare the issues with the .1mm and the .2mm sliced parts, the only similarity is the number of lines of code reached when trouble begins...I feel the .2mm makes it out of the woods by the skin of it's teeth, while the .1mm is only half way there.

[WarMachine]

I have been out of state for business related dealings and was slightly surprised that this topic died. After returning I sliced up yet another part with slightly over a million lines, and to no surprise for me, the dry run failed just as the others have.

I have more or less determined that my expectations for this setup exceeded the capabilities. I thank everyone for their input.

[Eaglezsoar]

Warmachine, there are some posts from Flateric where he is experiencing very similar problems as you.
If memory serves, one solution was to switch to Pronterface host for the Repetier firmware. Switching to
ASCII instead of Binary. The link is here: http://forum.seemecnc.com/viewtopic.php?f=37&t=2431