The above is just not done in Japan.
Perhaps we should differentiate between a tip and its poorer cousins; the round up, and the wave off. A tip is usually a type of reward for a service rendered. Given after the act, this is a common convention in many parts of the world. The round up is the act of giving more than the cost and is usually followed with the phrase, "Keep the change," or something similar. The 'wave off' is the declining of insignificant change from a purchase. Often this is done with a wave of the hand.
Tipping in Japan
Contrary to popular belief, there are times and places where tipping is practiced. Off hand, I can't think of instances where one would tip in the western fashion - after the service has been rendered. Generally, a tip, known as kokorozuke (心付け), is given before the service is performed. It has been described as being part of one of the greeting rituals. Usually, but not always, Kokorozuke money is given in a small envelope; not in the elaborate envelopes that are common at weddings or funerals. The kokorozuke envelopes are simple affairs, often on the small side.
The kokorozuke that most people are familiar with is the one used at upscale ryokans or onsens. This is given to the staff person, nakai-san (中居), who shows you to your room. This person will be responsible for one or more rooms and it's their job to set things up and put things away during your stay. Typically this person will show you to your room, set out the meal, clean away everything after the meal, and set out the bedding. They are the room's factotum.
I don't think kokorozuke at ryokans or onsens are necessarily expected, especially from a foreigner, but I'm sure they are appreciated. Should you decide you want to do this, it would be best to get a small envelope ready before hand. Handing over cash is not de rigueur. That said, I have seen it done. The bill, I think 5000¥, was folded over. I believe paper notes are folded in thirds. It would be presented when the nakai-san has finished getting you situated and is taking her leave.
Another common kokorozuke is during the run up to a wedding. Kokorozuke is often given to various people involved in the production of the event. I'm not familiar with the protocol, but I suspect that the hair dresser and ceremony hall people are amongst a long list of persons on the receiving end.
Other venues often get kokorozuke. If your large group is going to take over a restaurant or bar, it is not uncommon to give kokorozuke.
There is another kokorozuke that deserves mention. This is the one given to the guys doing the moving. Unlike the previously mentioned, the moving men kokorozuke is given after they start working, but before lunch. Sometime before lunch, a plain envelope with about 1000 yen (?) per mover is handed to the leader of the team while at the same time saying, "Use this for lunch or something."
Rounding Up in Japan
About the only time I've seen this used is with taxi drivers. It is not uncommon to give the taxi driver 3000 yen for a 2560 ride and tell him, "Change is unnecessary." A driver may baulk at receiving something substantially over the fare, but this is easily over come by telling the driver, "Okay, okay," and making a soft patting gesture while getting out.
The Wave Off in Japan
It is not uncommon in many countries to wave off the small change due at the end of a purchase. As others have mentioned, doing so in Japan can cause the cashier to chase you out of the establishment with your 'forgotten' 1 Yen.
It is probably a better practice to just accept the change gracefully. In convenience stores there is often a donation box on the counter. Should you really be averse to small change you can add it to the box.
Or you could just go wild and leave a really big tip.