A Case Study of Scalability Related "Out of memory" Crash in Erlang

We are building a platform for message switching, in Erlang. Everything looks OK on stability and features. It actually has run more than half year with zero down. We tested its performance on our 2-core CPU machine before, and got about 140 transactions/second, it's good enough.

Then, we got a 8-core CPU machine several weeks ago, and we did same performance testing on it, to see the scalability. Since Erlang is almost perfect on scalability, you can image the result, yes, about 700 transactions/second now, scaled almost linear. Until it crashed with "out of memory" when million hits processed.

It left a very big "erl_crash.dump" file there, I had to dig the issue. My first guess was, were some remote requests (access db, access remote web service etc) timeout but the process itself was not timeout yet, and cause more and more processes kept in VM?

A quick grep "=proc:" erl_crash.dump showed that the total number of processes was about 980, which was reasonable for our case.

So, which process ate so many memory? A quick grep "Stack+head" erl_crash.dump showed that there was indeed a process with 285082125 size of Stack+head there.

Following this clue, I caught this process:

=proc:<0.4.0>
State: Garbing
Name: error_logger
Spawned as: proc_lib:init_p/5
Last scheduled in for: io_lib_format:pad_char/2
Spawned by: <0.1.0>
Started: Sun Apr  1 01:21:50 2012
Message queue length: 2086029
Number of heap fragments: 1234053
Heap fragment data: 281266956
Link list: [<0.27.0>, <0.0.0>, {from,<0.42.0>,#Ref<0.0.0.88>}]
Reductions: 72745575
Stack+heap: 285082125
OldHeap: 47828850
Heap unused: 121777661
OldHeap unused: 47828850
Program counter: 0x0764c66c (io_lib_format:pad_char/2 + 4)
CP: 0x0764c1b4 (io_lib_format:collect_cseq/2 + 124)

This process was error_logger, which is from OTP/Erlang standard lib: error_logger, writing received messages to log file or tty. The typical usage is:

error_logger:info_msg("~p:~p " ++ Format, [?MODULE, ?LINE] ++ Data))

Which will format Data to a String according to the Format string, and write it to tty or log file.

The above case showed the message queue length of process "error_logger" had reached 1234053, and the Stack+heap was 285082125, about 272M size.

So the cause may be, that the message queue could not be processed in time, the messages were crowded in error_logger's process and finally caused "out of memory". The bottle-neck was that when error_logger tried to format the message to String, Erlang VM was weak on processing them, which seemed to need a lot of CPU cycles.

In my previous blog, I talked about Erlang is bad on massive text processing. Erlang processes String/Text via List, which is obvious bottle-neck in Erlang now, with Erlang is getting much and much popular and more and more Erlang applications are written.

But, why this did not happen on our 2-core CPU machine? It's an interesting scalability related problem:

"error_logger" module will registered one and only one process to receive and handle all log messages. But Erlang VM's scheduler can not distribute ONE process to use multiple CPUs' computing ability. In our 2-core machine, the whole ability is about 140 transactions/second, the one process of "error_logger" just happened to have the power to handle corresponding log messages in time. Under 8-core CPUs machine, our platform scales to handle 700 transactions/second, but there is still only one process of "error_logger", which can not use 8-core CPUs' ability at all, and finally fail on it.

Erlang treats every process fairly (although you can change the priority manually), we can do a simple/quick evaluation:

1. 2-Core machine, keeping hits at 140 trans/second:
The number of simultaneous processes will be about 200, each process shares the CPU cycles: 1/200 * 2 Core = 1%

2. 8-Core machine, keeping hits at 700 trans/second:
The number of simultaneous processes will be about 980, each process shares the CPU cycles: 1/980 * 8 Core = 0.82%

So, the CPU cycles shared by error_logger process actually not increases.

BTW, I think error_logger should cut its message queue when can not process them in time (disk IO may also be slower than receiving messages).

Comments

1. Geoff Cant -- 2009-01-01 16:00

Next time you have an erl_crash.dump file you might want to take a look at crashdump_viewer in the standard distribution. It parses the dump file and lets you examine all the processes (including message queue and process heap contents) from a handy web interface.

2. Caoyuan -- 2009-01-03 16:00

Geoff,

Thanks. But I got erl_crash.dump is not an Erlang crash dump

3. Cassandra -- 2012-02-28 19:14

Thanks for posting this, it was very informative and helped me very much.

Cassandra http://creditcard-ok.blogspot.com/

4. Messi Soccer sheos -- 2012-03-09 05:55

You can still find lots of individuals http://www.oaksunglassesale.com/oakley-juliet-c-11.html cannot personal one couple of the actual sneaker as well as be reluctant to get one match regarding themselves. Inside virtually people's brain

5. lokcuy -- 2012-03-13 22:44

There is clearly any lot of money to know about this specific membuat blog. Post consider you produced certain nice points within features also.

6. rojak -- 2012-03-20 14:28

Following examine a couple of template blogger with the weblog posts in your web site now, as well as I really just like ones manner of posting. My partner and i added it to help this bookmark web site listing as well as will likely be checking again quickly. Pls try this web page while properly as well as allow me to understand what you believe.

7. roty -- 2012-03-23 14:02

Post discovered your own blog site on the internet lowongan kerja in addition to verify a number of of one's earlier threads. Continue to keep up that very good run. Post just additional up your own Feed to my WINDOWS LIVE MESSENGER News Readership. In search of forward to reading more from you later on! …

8. louis vuitton uk -- 2012-03-31 05:07

Our online store offers you discounted Designer louis vuitton replica wallet at present. You could find them in desirable quality and price. If you don't mind high class louis vuitton uk, have a good time here.These is the first time your visit our louis vuit Louis vuitton online shop ,welcome.

http://www.louisvuittonukk.org.uk

9. anonymous -- 2012-03-31 05:07

any louis vuitton outlet New backpack features usa a function grownup overall look. Varied piece allows that it is captivated me within the approve and also further than your body.It’s induced by way of severe hardworking liver diseases, and also the most familiar factors behind constant louis vuitton bags outlet hard working liver disorder usually are excessive drinking and also liver disease Chemical.Ladies have an ardent love for the Louis Vuitton handbags outlet because Louis Vuitton enjoys a worldwide reputation of high quality and fashionable designs.

http://www.louisvuittonoutletbagsc.com

10. coach outlet store online -- 2012-03-31 05:07

coach outlet store online marketed properly all greater compared to earth and earn cozy praise from customers. They are made from the finest leather and fabric.coach outlet store have Coach handbags,Coach Shoulder Bags,Coach Briefcases and so on,these bags are so perfectly reproduced,you won't even be able to tell the difference!Coach Outlet Online Store would dynamically change your overall styles right away. The amazing knack about the unique coach handbag is that it would never disappoint your individual styles at all. Rather, it would instantly change your ultimate fashions in a remarkable manner.

http://www.coachoutletstoreonlineeo.com

11. coach factory outlet -- 2012-03-31 05:15

No one can deny the shopping at the coach factory outlet is satisfactory. For the low prices and good quality.Over the years, coach factory online has added a multitude of new handbag shapes, styles and materials to their collection. However, the highest care is taken that every Coach handbags is both aesthetically beautiful and functional.Many fashionable women match with practical Coach Purses which will make the street shopping become relaxed,and make every person can enjoy more diversiform combination in coach factory outlet online. http://www.ccoachfactoryoutlet.com

12. louis vuitton sale -- 2012-03-31 05:15

Don't feel upset, there will be a great conversion to this kind of situation because there are a large number of louis vuitton sale now!louis vuitton outlet,welcome to buy urban louis vuitton on our online shop.discount price is our special offer, durability and high quality is our promise.louis vuitton Outlets offer famous classic brand for LV,Channel, with perfect service.So become to the VIP soon.They offer more new styles,like LV purses,LV wallets etc. And are tested by product quality monitoring center .

http://www.louisvuittonoutletsaleo.com

13. coach outlet online -- 2012-03-31 05:15

Turn your attention to such discount coach sneakers for women from coach outlet online, you will find something unique and special of such authentic coach for sale at coach factory outlet store online.coach outlet store sells goods that are constructed to meet the highest standards of quality and functionality.You can trust it 100 percent.coach outlet has become a popular shopping experience for consumers around the world, and a desirable distribution channel for manufacturer's and retailers.

http://www.coachoutletonlinesl.com

14. nike TN -- 2012-03-31 09:10
15. rahul -- 2012-04-07 05:14

What an interesting info is visible in this blog that to very happy for the amazing technology is visible in this md5 decrypter blog. Thank you very much for provide the great info in this blog that to sharing the nice services in this blog.

16. chistian louboutin uk -- 2012-04-18 01:09
17. louis vuitton uk -- 2012-04-18 01:19
18. mulberry outlet uk -- 2012-04-18 01:24
19. website design bangalore -- 2012-04-27 07:13

I really enjoyed this site. This is such a Great resource that you are providing and you give it away for free. It gives in depth information. Thanks for this valuable information. http://www.nuvodev.com/

20. dave -- 2012-05-04 16:10

You can see the erl_crash.dump file at crushdump_viewer. Thanks for this informative turotial. lawdepot law depot

21. best skirts -- 2012-05-05 11:21

I am sure you hade a lot of fun writing this article! best skirts

22. http://www.nicerjerseys.net -- 2012-05-09 09:23

Hello, friends! Welcome to http://www.nicerjerseys.net . We offer the nicer jerseys, caps and so on. We have updated a lot of jerseys such as: NFL, MLB, NBA, women jerseys and youth jerseys. If you pay by Western Union , you’ll get 10% discount. We can give you the best discount. If you have any question please ask me or email me. Thanks a lot for looking around and ordering.

23. anonymous -- 2012-05-15 02:10

"Magic" 23 headphones by well-known hip-hop godfather, http://www.monsteroutletsaleclub.com/goods-11199-discount-cheap-monster-beats-by-dr-dre-solo-hd-headphones-black.html Monster Beats By Dr. Dre Solo HD Headphones Black Grammy winner Dr.Dre and Monster (Monster), developed a landmark headphones Monster Beats, the biggest feature of the headset is a strong bass, surging dynamic and extreme detail. The birth of monster of the headset, http://www.monsteroutletsaleclub.com/goods-11196-discount-cheap-monster-beats-by-dr-dre-solo-hd-headphones-red.html Monster Beats By Dr. Dre Solo HD Headphones Red set off a storm, sports and arts and culture in the United States, monster from the Exocet Phelps to Kobe Bryant, from the USA basketball dream team to all kinds of music MV, http://www.monsteroutletsaleclub.com/goods-11193-discount-cheap-monster-beats-by-dr-dre-solo-hd-headphones-black.html Monster Beats By Dr. Dre Solo HD Headphones Black can be seen in its presence. http://www.monsteroutletsaleclub.com

24. Beats Headphones -- 2012-05-18 02:32

http://www.monstercountryssale.com/goods-11157-discount-cheap-monster-beats-dr-dre-studio-classic-new-champagne-headphones-for-sale.html http://www.monstercountryssale.com/201203g11157-G-5709884/cheap-monster-beats-dr-dre-studio-classic-new-champagne-headphones-for-sale.jpg

A headset, http://www.monstercountryssale.com/goods-11149-discount-cheap-beats-dr-dre-studio-classic-james-limited-edition-gold-monster-headphonefor-sale.html Beats dr dre studio Classic James limited edition Gold Monster headphone design cool design for night elf style, gear like the outline of the ear shell design, a strong mechanical sense of the whole aural wear comfortable, customized create . http://www.monstercountryssale.com/goods-11148-discount-cheap-monster-beats-dr-dre-studio-classic-lamborghini-limited-edition-foe-sale.html Monster Beats Dr Dre Studio Classic Lamborghini Limited Edition broadband head for gamers The beam structure of double-layer design, the outer bracket, play a role in regulating the length of the first beam, a red leather inner, http://www.monstercountryssale.com/goods-11147-discount-cheap-monster-beats-by-dr-dre-studio-classic-red-sox-definition-headphones-for-sale.html Monster Beats by Dr. Dre Studio Classic Red Sox Definition Headphones elastic automatically adapt to different groups of people head against the head pressure to reduce. This two-tier structure of the first beam, excellent air permeability in the hot summer. http://www.monstercountryssale.com/

25. jual baju online -- 2012-05-19 09:53

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. <a href="http://bestcafeshops.com/jaketonline/jual-baju-online">jual baju online</a>

27. anonymous -- 2012-06-02 12:42

Make a cartoon character of a ferret doing kung fu in uniform in png format. decent rate

28. cheap handbags -- 2012-06-12 03:23

Designed with excellent craftsmanship, noble taste and latest style, every cheap handbag is hot selling with a large discount. With a great variety of <strong><a title="cheap handbags" href="http://www.handbags4discount.com">cheap handbags</a></strong> to choose from, you can certainly get your favorite cheap handbag of top quality and competitive price . Since we were engaged in selling <strong><a title="cheap handbags" href="http://www.handbags4discount.com">cheap handbags</a></strong>, we have paid much attention to both function and fashion of every handbag. Therefore, you can be assured that cheap handbag are convenient, trendy and attractive whenever and wherever.

29. John -- 2012-06-21 19:45

Hey, I think your really on track with this, I cant say I am completely on the same page, but its not really that big of a issue. department store credit cards

30. josephsholland -- 2012-07-03 16:54

I am very much 777 Best pokers happy for providing the good articles is Blackjack Online Play visible in this blog and using the great technology in this 888 POKER GAMES blog. Thanks a lot for providing the 777 Poker Online great services in this blog and Online top poker keep it up.

31. wmodica@… -- 2012-07-04 03:37

Denver Broncos,Every week he would rotate to a new fad diet, and my family ended. Authentic Nike NFL Jerseys| wholesale jerseys| cheap jerseys wholesale

32. nancyhanson -- 2012-07-13 15:36

This is really enjoyed with this info in this Online top poker blog that to helpful info in this blog. This 777 Poker Online website design is very different style that to this style is entirely different for the great technology in this 888 POKER GAMES blog. I am very much impressive for the info in this Blackjack Online Play blog. Thanks a lot for sharing the nice info in this blog that to using the great technology in this 777 Best pokers website also.

33. davidsfrost -- 2012-08-01 08:35

This is really admired for the great technology in this blog that to using the great technology in this blog. Thanks a lot for providing the nice services in this blog that to sharing the great technology in this blog. For more details visit our site ladbrokes bingo wink bingo sun bingo new bingo roulette table

34. davidsfrost -- 2012-08-02 06:27

This website is providing the superb info in this blog. I am really satisfied by the great technology is visible in this blog. Thanks a lot for providing the nice info is visible in this blog. For more details visit our site roulette table roulette Online cam roulette man roulette

35. davidsfrost -- 2012-08-03 12:08

Thanks a lot for providing the great technology is visible in this blog that to using the great services in this website. Thanks a lot for providing the great info is visible in this blog that to using the great technology. For more details visit our site russian roulette poker games free poker games gambling commission gambling

36. anonymous -- 2012-08-15 19:16

This is a nice post in an interesting line of content, great way of bring this topic to discussion. alabama auto insurance company

37. anonymous -- 2012-09-01 06:26

I really enjoyed reading this. pretty nice post, it is really kind of you to share with us. health insurance quotes nc

38. how to get a flat stomach for girls -- 2012-09-04 19:53

hey buddy, this is a very interesting article <a href="http://www.howtogetaflatstomachforgirls.com/" title="flat stomach for girls"><b>flat stomach for girls</b></a> <a href="http:/www.howtogetaflatstomachforgirls.com/flat-stomach-fast-for-girls/how-to-get-a-flat-stomach-for-girls-a//" title="stomach for girls"><b>how to Get flat stomach</b></a> <a href="http://www.fatburningfoodsformen1.com//" title="fat burning foods"><b>fat burning foods</b></a> <a href="http://pameungpeukselatan.blogspot.com/" title="pameungpeuk"><b>pameungpeuk</b></a> <a href="http://www.howtogetaflatstomachforgirls.com/abs/how-to-get-abs-for-girls-women//" title="abs for girls"><b>abs for girls</b></a>

40. anonymous -- 2012-09-16 16:45

This is also a very good post which I really enjoyed reading. It is not everyday that I have the possibility to see something like this <a href="http://kadalmesir.blogspot.com/2012/08/sepeda-motor-bebek-injeksi-kencang-dan.html">Sepeda Motor Bebek Injeksi Kencang dan Irit Jupiter Z1</a>

http://tinyurl.com/8e5h39y

41. anonymous -- 2012-09-17 00:31

<a href="http://www.iwinterparka.com">Canada Goose Parka</a><a href="http://www.iwinterparka.com">Canada Goose Jacket</a><a href="http://www.iwinterparka.com">Canada Goose Outlet</a>

42. longchamp outlet -- 2012-09-24 01:13

Yesterday in the Hankou Railway Station, http://www.longchampsoutletwholesales.com the first one to buy a men's Zheng Wu high-speed rail tickets. This reporter has learned, he called Zou Chang Jian, http://www.longchampsoutletwholesales.com intends to 28 by Zheng Wu high-speed rail travel to Zhengzhou. He said that by the first group of http://www.longchampsoutletwholesales.com high-speed rail to Zhengzhou, vacated more time for travel.

43. Sepeda Motor Bebek Injeksi Kencang dan Irit Jupiter Z1 -- 2012-09-24 05:54

hanks a lot for providing the great technology is visible in this blog that to using the great services in this website. Thanks a lot for providing the great info is visible in this blog that to using the great technology. For more details visit our site

<strong><a href="http://kadalmesir.blogspot.com/2012/09/sepedamotorbebekinjeksikencangdaniritju.html">Sepeda Motor Bebek Injeksi Kencang dan Irit Jupiter Z1</a></strong>

44. True Religion Jeans Outlet -- 2012-10-11 05:59

Serums and mild hair sprays square measure currently turning into one among the foremost essential beauty merchandise. If you've got long hair, you wish a product that http://www.truereligionjeansoutlet-onsale.com/ True Religion Outlet controls the kink up and keeps long hair stunning and glossy. girls with crisp hair may like a reliable product to take care of the curls and convey out nice volume and bounce. you'll use your Parlux http://clarisonicmia.brushstore0o.com/ Clarisonic Outlet hand blower reception, then bring a hair blood serum or mild spray for a fast fix.

45. anonymous -- 2012-10-12 01:17

Welcome!Coach Outlet Store are loved by many people,when you walk in the street, you could see many people take Coach Shoulder Bags styles.

Coach Factory http://coachfactoryoutlet.1uxury--best.com

Coach Factory Outlet http://www.coachfactoryoutletcd.com

Coach Outlet Store http://www.coachoutletstoreusac.com

Coach Outlet http://www.coachoutletonlinecss.com

47. Industrial Mixer -- 2012-10-15 01:49

The Grinding Machine in accordance with the type, divided into basket grinding machines, horizontal grinding machines, and rods nitrate mill. Many beginners may have lots of questions about the primary knowledge of selecting Industrial Mixer.

48. harry -- 2012-10-16 08:58

This article is exactly what I needed, thank you for sharing. I would recommend friends over to read. I hope you can keep writing more and more good articles. This is really useful to me. self drug test

50. Coach Outlet. -- 2012-10-23 04:51

http://www.coachoutletcheaper.net Coach Outlet http://www.coachoutletnow.net coachoutletnow.net http://www.coachoutletcheaper.net www.coachoutletcheaper.net http://www.coachoutletnow.net Coach Outlet Store Online http://www.coachoutletcheaper.net Coach Outlet Online Store The content is from Cherrychen 2012-10-23

51. anonymous -- 2012-12-13 05:55
52. wedding dresses 2013 -- 2012-12-27 07:42

So that you can attract http://www.diodress.net/cheap-bridesmaid-dresses-c8273/ cheap bridesmaid dresses far more shoppers, we add many new models to our cute party http://www.diodress.net/cheap-lace-wedding-dresses-c8258_8269/ lace wedding dresses . When you wear our cute get together dresses to the party, you will get several compliments from your people today. Cute get together dresses can make you extra eye-catching and beautiful. In addition to, we utilize the first-class elements to make the cute celebration dresses. Once you touch these cute get together http://www.diodress.net/cheap-halter-wedding-dresses-c8258_8267/ halter wedding dresses , you will really feel extremely at ease and soft.

54. anonymous -- 2013-01-17 15:49

http://www.airhighheels2u.com/ Nike Air Jordan 11 Heels

http://www.topbeats2013.com/ monster in-ear

http://www.warmmoncler2013.com/ moncler down coat women

http://www.warmjackets4u.com/ Moncler Men's Jackets

http://www.toplouboutin2u.com/ Christian Louboutin Sandals

55. sharathseo4@… -- 2013-01-22 17:08

Wow, this <a href="http://www.777PokerGames.com/">777 Poker Games</a>is really satisfied by the nice info is visible in this blog that to <a href="http://www.BlackjackPlayLive.com/">Blackjack Play Live</a> using the excellent science is visible in this <a href="http://www.365CASINOGAME.COM/">365 CASINO GAME</a> blog. This web portal is providing the hard to believe technology is visible in this blog that to using the nice services in <a href="http://www.CasinoGamePlan.com/">Casino Game Plan</a> this blog. I had really premier info is visible in this blog <a href="http://www.LiveCasinoSport.com/">Live Casino Sport</a> that to using the exceptionally awesome technology is visible in this blog. Thanks a lot for providing the nice services in this blog.

56. anonymous -- 2013-01-24 01:48

Several buses are available near Potong Pasir Sant Ritz along with shopping centers and restaurants. Sant Ritz is also near to Volkswagen Golf Centre and Kallang Side Riverpark. Entertainment for your loved ones and friends is therefore at your fingertips with the full condo facilities as well as the amenities near Potong Pasir.

57. anonymous -- 2013-03-02 03:23

Great Post. I have not been visiting the site recently. Took a visit again and there were some great comments on the site. Excellent post. Keep up the good work. Bartley Ridge

58. jerseysshow@… -- 2013-03-13 02:06

Popular jerseys wholesale welcome to: http://www.jerseysshow.me/ Offer cheap, high quality popular jerseys (Nike NFL,NBA Colleage Basketball…), all names & numbers are sewn on. if you order 10 or more items, free shipping . Pay by western union ,you will get a 10% discount. Looking forward to your visiting.

59. Toms Outlet -- 2013-03-21 08:17

will start to feel more normal," Crowe says.New http://www.tomsshoesonsaleforcheap.com/ home sales are especially important to the economy because buyers then spend money on a raft of other items, such as furnishings,

60. movies on line -- 2013-03-30 14:18

"It takes a great deal of bravery to stand up to your enemies, but a great deal more to stand up to your friends." <a href="http://movies-digital.com/">movies on line</a>

62. free movies online -- 2013-04-04 12:12

I love you. I am who I am because of you. You are every reason, every hope, and every dream I've ever had, and no matter what happens to us in the future, everyday we are together is the greatest day of my life. I will always be yours. .<a href="http://movies-digital.com/">free movies online</a>

63. free movies online -- 2013-04-04 12:24

I love you. I am who I am because of you. You are every reason, every hope, and every dream I've ever had, and no matter what happens to us in the future, everyday we are together is the greatest day of my life. I will always be yours. .<a href="http://movies-digital.com/">free movies online</a>

64. movies on demand -- 2013-04-07 09:43

"Oh yes, the past can hurt. But you can either run from it, or learn from it."<a href=" http://movies-digital.com/">movies on demand</a>

65. movies online -- 2013-04-08 11:02

The truth is... I gave my heart away a long time ago, my whole heart... and I never really got it back.. <a href="http://movies-digital.com/">movies online</a>

68. rent movies online -- 2013-04-09 10:47

The best love is the kind that awakens the soul and makes us reach for more, that plants a fire in our hearts and brings peace to our minds. And that's what you've given me. That's what I'd hoped to give you forever. . <a href="http://movies-digital.com/">rent movies online</a>

69. free movies -- 2013-04-10 09:14

Why are you trying so hard to fit in when you were born to stand out?. <a href=" http://movies-digital.com/">free movies</a>

70. anonymous -- 2013-04-16 00:14

Welcome to: http://www.likejerseys.net/ LikeJerseys? in the export of jerseys and sell nfl jerseys,nhl jerseys,mlb jerseys,nba jerseys,sunglass,t-shirt,handbag and so on. we rely on the outstanding service and high quality obtain high reputation in USA ,canada. And most products displayed on our website are available and have stocks here.

72. anonymous -- 2013-04-17 12:14

Belgravia Villas is a new and upcoming cluster housing located in the Ang Mo Kio area, nested right in the Ang Mo Kio landed area. It is within a short drive to Little India, Orchard and city area. With expected completion in mid 2016, it comprises of 118 units in total with 100 units of terrace and 18 units of Semi-D. http://belgravia-villa.com.sg

73. anonymous -- 2013-04-22 11:23

Ecopolitan EC has full and unique facilities, which includes a guard house, clubhouse, Function Room & Indoor Gym Tennis Court, 50m Freeform Pool Pool Deck, Wading Pool, Splash Pool & Family Pool Jacuzzi & Hydro Spa, BBQ Area Dining and Play Fountain, Fitness Alcove & Children’s Playground and Garden Trail. The condo’s facilities provide full family entertainment needs for your family and loved ones. Indulge in a serene and tranquil lifestyle right in the heart of Punggol. Ecopolitan

74. anonymous -- 2013-04-28 01:07

​@linlinhttp://jstlouisvuitton.webs.com/ [Louis Vuitton Outlet] ​http://christianlouboutinblogss.webs.com/ [Christian Louboutin Sale] ​http://jstjordanair.webs.com/ [Authentic Air Jordan Shoes]

75. anonymous -- 2013-05-03 07:00

Excellent Post. Thank you Author for such a wonderful write up. Indeed it was very detailed. I have bookmarked your hillview blog. Thanks Hillview Peak

76. Devid778@… -- 2013-05-10 08:11

Awesome Case study, Learn a lot from it, Thanks http://weezlabs.com

77. anonymous -- 2013-05-13 08:25

Future residents will be able to access the development via Buangkok MRT which is just next to it. Also, nature awaits your family and friends at the nearby Punggol Park. Also, the ultimate nature awaits you at the Sengkang Riverside Park. Jewel at Buangkok

78. anonymous -- 2013-05-14 04:09

Future residents will be able to walk to Jurong East MRT Station which is located right beside it. Also, nature awaits your family and friends at the Jurong Lake Park and the Jurong Country Park. Also, the ultimate nature awaits you the Japanese Garden. J Gateway

79. anonymous -- 2013-05-22 06:25
82. anonymous -- 2013-05-31 01:58

Coral Edge Residences has full and unique facilities, which includes a guard house, clubhouse, Function Room & Indoor Gym Tennis Court, 50m Freeform Pool Pool Deck, Wading Pool, Splash Pool & Family Pool Jacuzzi & Hydro Spa, BBQ Area Dining and Play Fountain, Fitness Alcove & Children’s Playground and Garden Trail. The condo’s facilities provide full family entertainment needs for your family and loved ones. Indulge in a serene and tranquil lifestyle right in the heart of Punggol. Thx Coral Edge Residences

83. anonymous -- 2013-06-04 01:14

Future residents will be able to walk to the existing Bugis MRT. With such a short drive to the city area as well as the orchard and bugis area, entertainment for your love ones and family will come at a stone’s throw away. DUO Residences

84. anonymous -- 2013-06-09 11:26

Bartley Ridge has full and unique facilities, which includes a guard house, clubhouse, children’s playground, swimming pool, piano room, pool room, indoor gym, hydrotherapy beds, hydrotherapy baths, reading room, function room, onsen, jacuzzi. Bartley Ridge

85. anonymous -- 2013-06-09 11:27

Great Post. I have not been visiting the site recently. Took a visit again and there were some great comments on the site. Excellent post. Keep up the good work. Bartley Ridge

86. anonymous -- 2013-06-14 01:52

Lush Acres EC is a 99-years leasehold Punggol EC development located Sengkang West Way / Fernvale Link in District 19. With expected completion in mid 2016, it comprises of TBA towers with TBA units and stands TBA storeys tall. It is situated right beside Kupang LRT Station. Lush Acres

87. anonymous -- 2013-06-14 01:59

Great Post. I have not been visiting the site recently. Took a visit again and there were some great comments on the site. Excellent post. Keep up the good work. Lush Acres

88. anonymous -- 2013-06-20 23:34

In advance of buying your own pair of trainers, ask you that what amount time daily you shell out while going or wandering, if the right formula is amongst 25 mints in an hour as compared with an inexpensive pair with puma shoes do, but when you've got walk distance and distance than go with much improved quality.

Believe people shopkeeper would not feel upset in expressing you all the range with shoes for ladies they are obtainable http://www.abcsnk.com アディダスシューズ with, therefore take a look at thoroughly before purchasing one http://www.abcsnk.com/adidas-スパイクサッカー-f50-アディゼロ-c-22_23_24.html f50 スパイク for you, as just simply the name will likely not http://www.sportrakuten.com/categories/1369381929-1.html new ERA ニューエラcarharnew do it's the comfort they can be offering that creates the entire difference since you will walk with them (and anybody have their standards with comfort).

Never pass what others have got to say for your particular make, there will be chances this adidas shoes are usually more suitable to the friend rather than pair with puma boots and shoes, always remember you may have your mind to implement therefore go with one that suites you actually perfectly never your companion.

Last yet not the least go with comfort covering the style, I recognize style plus fashion trend is critical but that is definitely something you may follow when http://www.abcsnk.com/adidasアディダス-adidas-スパイクサッカー-c-22_23.html アディダスサッカースパイク purchasing a person's catwalk boots and shoes, while shopping for http://www.sportrakuten.com ランニングシューズ通販 your trainers give extra points to your comfort when compared to http://www.sportrakuten.com/categories/1369390315-177.html オークリー (OAKLEY) style. Always remember it's important to walk 100 % day dressed in them, hence never endanger on relaxation for design and style.

89. anonymous -- 2013-06-21 05:35

It is a short drive away from Woodlands and Admiralty MRT Station. Future residents will be able to walk to the nearby Vista Point or a short drive to Causeway Point for some family fun and gatherings. A truly unique lifestyle awaits you. Forestville EC

90. anonymous -- 2013-06-26 16:19

Id really love to stay around this website and be a part of where I can get a lot of valuable and beneficial advice from other experienced people that share this information. Duo Residences J Gateway

91. anonymous -- 2013-06-28 02:27

great blog. [htp://topiarysg.com/ Topiary] [htp://topiarysg.com/ Topiary EC] [htp://topiarysg.com/ Topiary Executive Condo] [htp://topiarysg.com/ Topiary Executive Condominium] Twin Fountains Twin Fountains Executive Condo Twin Fountains Woodlands Twin Fountains Executive Condominium

93. anonymous -- 2013-07-02 02:03

Future residents will be able to walk to the existing Bartley MRT in the Circle Line. With such a short drive to the city area as well as the orchard and bugis area, entertainment for your love ones and family will come at a stone’s throw away. Kensington Square

94. anonymous -- 2013-07-06 08:11
95. anonymous -- 2013-07-06 08:12

New Singapore executive condo at Pasi ris! visit http://sea-horizon-ec.org or http://seahorizon-ec.biz

96. anonymous -- 2013-07-06 08:13

Best new condo in Singapore! Visit http://singaporecondo.co Look out for latest news!

97. anonymous -- 2013-07-08 05:14

Future residents will be able to walk to the nearby Sun Plaza or a short drive to Causeway Point for some family fun and gatherings. A truly unique lifestyle awaits you at Skypark Residences. Skypark Residences

98. anonymous -- 2013-07-08 05:22

Sembawang EC has full and unique facilities, which includes a guard house, clubhouse, Function Room & Indoor Gym Tennis Court, 50m Freeform Pool Pool Deck, Wading Pool, Splash Pool & Family Pool Jacuzzi & Hydro Spa, BBQ Area Dining and Play Fountain, Fitness Alcove & Children’s Playground and Garden Trail. The condo’s facilities provide full family entertainment needs for your family and loved ones. Indulge in a serene and tranquil lifestyle right in the heart of Woodlands. Sembawang EC

99. anonymous -- 2013-07-10 00:13

rare freehold condominium. check it out at Tembusu Tembusu Condo Tembusu Condominium Tembusu at Kovan

100. anonymous -- 2013-07-12 05:59

With expected completion in mid 2016, Sea Horizon EC comprises of 13 towers with 495 units and stands 13 storeys tall. Future residents will be able to access the development via Pasir Ris MRT. Sea Horizon

101. anonymous -- 2013-07-12 06:08

Several buses are available near Pasir Ris EC along with shopping centers and restaurants. Pasir Ris EC is also near to Downtown East, the ultimate entertainment centre in the East. Entertainment for your loved ones and friends is therefore at your fingertips with the full condo facilities as well as the amenities near Sea Horizon EC. Pasir Ris EC

102. anonymous -- 2013-07-13 09:04

Future residents will be able to walk to the existing Tanah Merah MRT station. Also, nature and recreation awaits your family and friends at the nearby Tanah Merah Country Club and the National Sailing Centre. Also, the ultimate nature awaits you at the East Coast Parkway which is just a short drive away from The Glades The Glades

103. anonymous -- 2013-07-17 10:36

It shows how well you understand this subject. Bookmarked this page, will come back for more. The Glades

104. anonymous -- 2013-07-17 10:38

Excellent information on your blog, thank you for taking the time to share with us. Amazing insight you have on this, it's nice to find a website that details so much information about different artists. The Skywoods Kensington Square

106. Pasir Ris Rise EC -- 2013-07-26 01:58

Several buses are available near Sea Horizon EC along with shopping centers and restaurants. Sea Horizon EC is also near to Downtown East, the ultimate entertainment centre in the East. Entertainment for your loved ones and friends is therefore at your fingertips with the full condo facilities as well as the amenities near Sea Horizon EC. Sea Horizon

107. http://www.beabingo.net -- 2013-07-30 07:54

I just stumbled upon your weblog and wished to say that I have truly enjoyed surfing around your blog posts. After all I’ll be subscribing to your rss feed and I hope you write again soon! http://www.beabingo.net

108. The Skywoods -- 2013-07-31 06:30

Just across your blog and i find it really interesting. Been reading some of your blogs too. Keep it up!

The Skywoods @ Dairy Farm Road

109. anonymous -- 2013-07-31 06:53
110. waterwoods ec punggol -- 2013-07-31 09:36

Developed by renowned Sing Holdings, Waterwoods is located at the junction of Punggol Field Walk and Punggol East and is just a 5 mins walk to Coral Edge LRT and Punggol Plaza. waterwoods ec punggol

111. sea horizon pasir ris -- 2013-07-31 09:36

Residents at the Sea Horizon can get to nearby supermarkets and shopping malls for a variety of amenities which include grocery and retail shopping, bank, eateries and more! E!Hub is just a 5 mins walk from this development while White Sands at Pasir Ris MRT Station is just a short drive away! sea horizon pasir ris

112. skypark singapore -- 2013-07-31 09:37

The first ever luxurious executive condominium is launching soon in the maturing estate of Sembawang. This EC come with full condo facilities, with modern design architecture, and the finest interior finishing to be expected. This new Sembawang EC is within the close proximity to various amenities, and also just a stone throw away to Sembawang MRT station. skypark singapore

113. Daniel_R -- 2013-08-02 05:34

Thomson Three 20 storey wonder will rise only 300 meters away from the upcoming Upper Thomson MRT Station.

Three Singapore

114. EMMAER -- 2013-08-06 12:59

The Yuan deputy commissioner draws a deep sigh to say:"If the mother is like when you are so big but revolts like this, don't listen to parental, don't want to marry ten how old medical college

teachers of big oneself, result ……although your father is a good person, if can re- choose once, I will listen to the parents' guidance, small Fei, the mother is your most close person,http://www.newnike-airmax2013.co.uk do I how can harm you, everything done by me, all for the sake of how are you, the woman married a wrong person to finish during a lifetime, mama doesn't hope old road that your repeated mama once walks."http://www.newnike-airmax2013.co.uk

115. The Glades Singaproe -- 2013-08-14 16:32

<a href="http://http://thegladessingapore.com">The Glades Singapore</a> <a href="http://thegladessingapore.com/details">Details</a> <a href="http://thegladessingapore.com/why-the-glades">Why the glades</a> <a href="http://thegladessingapore.com/happenings">Happenings</a>

116. Singapore EC -- 2013-08-14 16:34

New Executive Condo in Singapore http://singaporeec.com

117. anonymous -- 2013-08-14 16:35

The Glades at Tanah Merah in Singapore! http://thegladessingapore.com

118. anonymous -- 2013-08-14 16:36

The Glades Condo http://thegladescondo.org

119. anonymous -- 2013-08-14 16:37

Glades COndo in Singapore! http://gladescondo.org

120. Singapore Condo -- 2013-08-14 16:38

Best Singapore Condo http://singaporecondo.co

121. anonymous -- 2013-08-31 06:21

<a href="http://anekabajubatik.com">batik murah</a> wow i didnt see that coming

123. anonymous -- 2013-09-15 02:18

Future residents are within walking distance to Bishan Junction 8 and a short drive to Ang Mo Kio Hub from Sky Vue Condo. With such a short drive to the city area as well as the orchard and bugis area, entertainment for your love ones and family will come at a stone’s throw away from Sky Vue Condo <a href="http://sky-vue.com.sg">Sky Vue</a>

124. inflora -- 2013-09-19 13:35

<a href="http://inflorasingapore.org">inflora singapore</a> <a href="http://inflorasingapore.org">inflora condo</a> <a href="http://inflorasingapore.org">inflora</a> <a href="http://inflorasingapore.org">inflora condo singapore</a>

125. danaray -- 2013-09-29 14:03

I am impressed from the article of this Team about new plain. They are doing good job for people. I like it. Keep it up... <a href="http://www.99writing.com/essay-writing/">Essay Writing</a>

126. Princess -- 2013-10-01 10:51

LucidIre Community Ltd. is the best community i have joined for this year. I have enjoy different threads and informative sharing. Visit them too :)

127. anonymous -- 2013-10-07 08:44

The Hillford Condo has full and unique facilities, which includes a guard house, clubhouse, children's playground, swimming pool, Aerobic/Yoga? room, piano room, pool room, indoor gym, hydrotherapy beds, hydrotherapy baths, reading room, function room, onsen, jacuzzi. <a href="http://thehillford.com">The Hillford</a>

128. anonymous -- 2013-10-09 00:02

With expected completion in mid 2017, Jurong West Condo comprises of TBA towers with TBA units and stands TBA storeys tall. It is just a stone throw away from Lakeside MRT Station. Jurong West New Launch by MCL Land

129. anonymous -- 2013-10-09 00:07

New Condo in Jurong is a 99-years leasehold Jurong West New Launch Condo development located at Jurong West Street 41. Jurong West New Condo project name

130. anonymous -- 2013-11-06 07:16

Different dogs require different forms of dog grooming. Also, the length of each grooming is different across different breeds. Also, across different breeds, the instruments needed to groom the dog will be different. http://dog-grooming.org

131. mesutfozilsiddiqreal@… -- 2013-11-12 04:49

I have seen your awareness about this theme when you post it and it really gives an informational message to us readers. I am hoping that you will continue writing this kind <a href="http://jwrecovery.org">jaket korea</a> http://jualjaket.co

132. mesutfozilsiddiqreal@… -- 2013-11-14 06:34

Making clear the distinction between performance as action and performance as act-ual production is very important indeed for goood outcomes.. . re- "Performance" as action see also Brissett & Edgley Life As Theater http://zslygame.com

133. mesutfozilsiddiqreal@… -- 2013-11-14 06:34

I just wanna say thank you for the information that you have been shared to us readers. Thanks for posting this kind of theme. http://gynexinreviewsproduct.com

134. anonymous -- 2013-11-23 06:44

Riverbank at Fernvale has full and unique facilities, which includes a guard house, clubhouse, Function Room & Indoor Gym Tennis Court, 50m Freeform Pool Pool Deck, Wading Pool, Splash Pool & Family Pool Jacuzzi & Hydro Spa, BBQ Area Dining and Play Fountain, Fitness Alcove & Children’s Playground and Garden Trail. Sengkang West Condo

135. anonymous -- 2013-11-23 06:49

Several buses are available near Sengkang West Way Condo along with shopping centers and restaurants. Sengkang West Way Condo is also near Waterway Point, the shopping, dining and entertainment hub which is scheduled to open in 2 years time. Also, it is right beside Punggol Waterfront. Entertainment for your loved ones and friends are therefore at your fingertips with the full condo facilities as well as the amenities near Sengkang West Way Condo. Riverbank at Fernvale