Went ahead and removed the X ends and redid them with 2 layers of foam tape. Seems to be working great.
To make things work, trimmed the width off 2 foam tape squares, cut a hole for the screw using an x-acto knife, and glued them together. Attached one of the acrylic rail sliders to the X end with the foam tape and a screw to hold everything in place. Tightened the screw carefully to compress the foam tape layers and then slide everything (X end plus both rail sliders) onto the Z rail. Once on the rail, added the screw to the second acrylic rail slider and loosened the screw on the foam tape slider. Was able to slide everything on and get a nice tight fit.
If you can push hard enough, you probably don't need this screw trick. In my case, this was the best option.