Enies Lobby is where the pacing starts to fall off in the anime, yeah. Probably a good time to move to the manga. Though the next arc has a certain song that absolutely needs to be listened to, so maybe watch the anime for it too.
Also the point of Lucci fight is that the guy is strong enough to kill anyone in the SH crew, and considering how tired they are, probably the whole crew combined. If Luffy doesn't beat him, they are dead.
I get that in theory, it just felt a bit... disconnected from the narrative somehow. Like the actual objective of them being there was achieved by everyone else, and Luffy was functionally acting as a distraction for Lucci. I guess that's fine, it was just strange.