Friday, February 17, 2017

Stingray Renderer Walkthrough #5: RenderDevice

Stingray Renderer Walkthrough #5: RenderDevice

Overview

The RenderDevice is essentially our abstraction layer for platform specific rendering APIs. It is implemented as an abstract base class that various rendering back-ends (D3D11, D3D12, OGL, Metal, GNM, etc.) implement.

The RenderDevice has a bunch of helper functions for initializing/shutting down the graphics APIs, creating/destroying swap chains, etc. All of which are fairly straightforward so I won’t cover them in this post, instead I will put my focus on the two dispatch functions consuming RenderResourceContexts and RenderContexts:


class RenderDevice {
public: 
    virtual void dispatch(uint32_t n_contexts, RenderResourceContext **rrc, 
        uint32_t gpu_affinity_mask = RenderContext::GPU_DEFAULT) = 0;

    virtual void dispatch(uint32_t n_contexts, RenderContext **rc, 
        uint32_t gpu_affinity_mask = RenderContext::GPU_DEFAULT) = 0;
};

Resource Management

As covered in the post about RenderResourceContexts, they provide a free-threaded interface for allocating and deallocating GPU resources. However, it is not until the user has called RenderDevice::dispatch() handing over the RenderResourceContexts as their representation gets created on the RenderDevice side.

All implementations of a RenderDevice have some form of resource management that deals with creating, updating and destroying of the graphics API specific representations of resources. Typically we track the state of all various types of resources in a single struct, here’s a stripped down example from the DX12 RenderDevice implementation called D3D12ResourceContext:


struct D3D12VertexBuffer
{
    D3D12_VERTEX_BUFFER_VIEW view;
    uint32_t allocation_index;
    int32_t size;
};

struct D3D12IndexBuffer
{
    D3D12_INDEX_BUFFER_VIEW view;
    uint32_t allocation_index;
    int32_t size;
};

struct D3D12ResourceContext 
{
    Array<D3D12VertexBuffer> vertex_buffers;
    Array<uint32_t> unused_vertex_buffers;

    Array<D3D12IndexBuffer> index_buffers;
    Array<uint32_t> unused_index_buffers;

    // .. lots of other resources

    Array<uint32_t> resource_lut;
};

As you might remember, the linking between the engine representation and the RenderDevice representation is done using the RenderResource::render_resource_handle. It encodes both the type of the resource as well as a handle. The resource_lut is an indirection to go from the engine handle to a local index for a specific type (e.g vertex_buffers or index_buffers in the sample above). We also track freed indices for each type (e.g. unused_vertex_buffers) to simplify recycling of slots.

The implementation of the dispatch function is fairly straight forward. We simply iterate over all the RenderResourceContexts and for each context iterate over its commands and either allocate or deallocate resources in the D3D12ResourceContext. It is important to note that this is a synchronous operation, nothing else is peeking or poking on the D3D12ResourceContext when the dispatch of RenderResourceContexts is happening, which makes our life a lot easier.

Unfortunately that isn’t the case when we dispatch RenderContexts as in that case we want to go wide (i.e. forking the workload and process it using multiple worker threads) when translating the commands into API calls. While we don’t allow allocating and deallocating new resources from the RenderContexts we do allow updating them which mutates the state of the RenderDevice representations (e.g. a D3D12VertexBuffer).

At the moment our solution for this isn’t very nice, basically we don’t allow asynchronous updates for anything else than DYNAMIC buffers. UPDATABLE buffers are always updated serially before we kick the worker threads no matter what their sort_key is. All worker threads access resources through their own copy of something we call a ResourceAccessor, it is responsible for tracking the worker threads state of dynamic buffers (among other things). In the future I think we probably should generalize this and treat UPDATABLE buffers in a similar way.

(Note: this limitation doesn’t mean you can’t update an UPDATABLE buffer more than once per frame, it simply means you cannot update it more than once per dispatch).

Shaders

Resources in the D3D12ResourceContext are typically buffers. One exception that stands out is the RenderDevice representation of a “shader”. A “shader” on the RenderDevice side maps to a ShaderTemplate::Context on the engine side, or what I guess we could call a multi-pass shader. Here’s some pseudo code:


struct ShaderPass
{
    struct ShaderProgram
    {
        Array<uint8_t> bytecode;
        struct ConstantBufferBindInfo;
        struct ResourceBindInfo;
        struct SamplerBindInfo;
    };
    ShaderProgram vertex_shader;
    ShaderProgram domain_shader;
    ShaderProgram hull_shader;
    ShaderProgram geometry_shader;
    ShaderProgram pixel_shader;
    ShaderProgram compute_shader;

    struct RenderStates;
};

struct Shader
{
    Vector<ShaderPass> passes;
    enum SortMode { IMMADIATE, DEFERRED };
    uint32_t sort_mode;
};

The pseudo code above is essentially the RenderDevice representation of a shader that we serialize to disk during data compilation. From that we can create all the necessary graphics API specific objects expressing an executable shader together with its various state blocks (Rasterizer, Depth Stencil, Blend, etc.).

As discussed in the last post the sort_key encodes the shader pass index. Using Shader::sort_mode, we know which bit range to extract from the sort_key as pass index, which we then use to look up the ShaderPass from Shader::passes. A ShaderPass contains one ShaderProgram per active shader stage and each ShaderProgram contains the byte code for the shader to compile as well as “bind info” for various resources that the shader wants as input.

We will look at this in a bit more detail in the post about “Shaders & Materials”, for now I just wanted to familiarize you with the concept.

Render Context translation

Let’s move on and look at the dispatch for translating RenderContexts into graphics API calls:

class RenderDevice {
public: 
    virtual void dispatch(uint32_t n_contexts, RenderContext **rc, 
        uint32_t gpu_affinity_mask = RenderContext::GPU_DEFAULT) = 0;
};

The first thing all RenderDevice implementation do when receiving a bunch of RenderContexts is to merge and sort their Commands. All implementations share the same code for doing this:

void prepare_command_list(RenderContext::Commands &output, unsigned n_contexts, RenderContext **contexts);

This function basically just takes the RenderContext::Commands from all RenderContexts and merges them into a new array, runs a stable radix sort, and returns the sorted commands in output. To avoid memory allocations the RenderDevice implementation owns the memory of the output buffer.

Now we have all the commands nicely sorted based on their sort_key. Next step is to do the actual translation of the data referenced by the commands into graphics API calls. I will explain this process with the assumption that we are running on a graphics API that allows us to build graphics API command lists in parallel (e.g. DX12, GNM, Vulkan, Metal), as that feels most relevant in 2017.

Before we start figuring out our per thread workloads for going wide, we have one more thing to do; “instance merging”.

Instance Merging

I’ve mentioned the idea behind instance merging before [1,2], basically we want to try to reduce the number of RenderJobPackages (i.e. draw calls) by identifying packages that are similar enough to be merged. In Stingray “similar enough” basically means that they must have identical inputs to the input assembler as well as identical resources bound to all shader stages, the only thing that is allowed to differ are constant buffer variables. (Note: by todays standards this can be considered a bit old school, new graphics APIs and hardware allows to tackle this problem more aggressively using “bindless” concepts. )

The way it works is by filtering out ranges of RenderContexts::Commands where the “instance bit” of the sort_key is set and all bits above the instance bit are identical. Then for each of those ranges we fork and go wide to analyze the actual RenderJobPackage data to see if the instance_hash and the shader are the same, and if so we know its safe to merge them.

The actual merge is done by extracting the instance specific constants (these are tagged by the shader author) from the constant buffers and propagating them into a dynamic RawBuffer that gets bound as input to the vertex shader.

Depending on how the scene is constructed, instance merging can significantly reduce the number of draw calls needed to render the final scene. The instance merger in itself is not graphics API specific and is isolated in its own system, it just happens to be the responsibility of the RenderDevice to call it. The interface looks like this:

namespace instance_merger {

struct ProcessMergedCommandsResult
{
    uint32_t n_instances;
    uint32_t instanced_batches;
    uint32_t instance_buffer_size;
};

ProcessMergedCommandsResult process_merged_commands(Merger &instance_merger, 
    RenderContext::Commands &merged_commands);

}

Pass in a reference to the sorted RenderContext::Commands in merged_commands and after the instance merger is done running you hopefully have fewer commands in the array. :)

You could argue that merging, sorting and instance merging should all happen before we enter the world of the RenderDevice. I wouldn’t argue against that.

Prepare workloads

Last step before we can start translating our commands into state / draw / dispatch calls is to split the workload into reasonable chunks and prepare the execution contexts for our worker threads.

Typically we just divide the number of RenderContext::Commands we have to process with the number of worker threads we have available. We don’t care about the type of different commands we will be processing and trying to load balance differently. The reasoning behind this is that we anticipate that draw calls will always represent the bulk of the commands and the rest of the commands can be considered as unavoidable “noise”. We do, however, make sure that we don’t do less than x-number of commands per worker threads, where x can differ a bit depending on platform but is usually ~128.

For each execution context we create a ResourceAccessors (described above) as well as make sure we have the correct state setup in terms of bound render targets and similar. To do this we are stuck with having to do a synchronous serial sweep over all the commands to find bigger state changing commands (such as RenderContext::set_render_target).

This is where the Command::command_flags bit-flag comes into play, instead of having to jump around in memory to figure out what type of command the Command::head points to, we put some hinting about the type in the Command::command_flags, like for example if it is a “state command”. This way the serial sweep doesn’t become very costly even when dealing with large number of commands. During this sweep we also deal with updating of UPDATABLE resources, and on newer graphics APIs we track fences (discussed in the post about Render Contexts).

The last thing we do is to set up the execution contexts with create graphics API specific representations of command lists (e.g. ID3D12GraphicsCommandList in DX12),

Translation

When getting to this point doing the actual translation is fairly straight forward. Within each worker thread we simply loop over its dedicated range of commands, fetch its data from Command::head and generate any number of API specific commands necessary based on the type of command.

For a RenderJobPackage representing a draw call it involves:

  • Look up the correct shader pass and, unless already bound, bind all active shader stages
  • Look up the state blocks (Rasterizer, Depth stencil, Blending, etc.) from the shader and bind them unless already bound
  • Look up and bind the resources for each shader stage using the RenderResource::render_resource_handle translated through the D3D12ResourceAccessor
  • Setup the input assembler by looping over the RenderResource::render_resource_handles pointed to by the RenderJobPackage::resource_offset and translated through the D3D12ResourceAccessor
  • Bind and potentially update constant buffers
  • Issue the draw call

The execution contexts also holds most-recently-used caches to avoid unnecessary binds of resources/shaders/states etc.

Note: In DX12 we also track where resource barriers are needed during this stage. After all worker threads are done we might also end up having to inject further resource barriers between the command lists generated by the worker threads. We have ideas on how to improve on this by doing at least parts of this tracking when building the RenderContexts but haven’t gotten around looking into it yet.

Execute

When the translation is done we pass the resulting command lists to the correct queues for execution.

Note: In DX12 this is a bit more complicated as we have to interleave signaling / waiting on fences between command list execution (ExecuteCommandList).

Next up

I’ve deliberately not dived into too much details in this post to make it a bit easier to digest. I think I’ve manage to cover the overall design of a RenderDevice though, enough to make it easier for people diving into the code for the first time.

With this post we’ve reached half-way through this series, we have covered the “low-level” aspects of the Stingray rendering architecture. As of next post we will start looking at more high-level stuff, starting with the RenderInterface which is the main interface for other threads to talk with the renderer.

144 comments:

  1. Very interesting series, thanks a lot!

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Your Shader::SortMode-enum has a typo in it.
    It should be IMMEDIATE, not IMMADIATE.

    ReplyDelete
  4. Top airlines in the world
    An airline is an organization that gives air transport administrations to voyaging travelers and cargo. Carriers use flying machine to supply these administrations, and may frame organizations or coalitions with different airlines for codeshare understandings. For the most part, airline organizations are perceived with an air working testament or permit issued by a legislative aeronautics body
    Visit for more :-Qantas Airlines Phone Number

    ReplyDelete
  5. PosLaju parcel tracker of the Malaysia & World. Add tracking number to track your PosLaju packages as well as obtain delivery status online.
    https://poslajutracking.xyz/
    poslaju tracking
    poslaju track and trace
    poslaju tracking number
    poslaju tracking express

    ReplyDelete
  6. This is a great article, with lots of information in it, These types of articles interest users in your site. Please continue to share more interesting articles!

    ReplyDelete
  7. Among, Infrastructure as a Service (IaaS), Software as a Service (SaaS), and Platform as a Service (PaaS), AWS chooses right kind of distributed computing and gives it to the business. AWS is known for adaptability with recognizable design, databases, operating systems, and programming dialects. It likewise guarantees security for the framework including physical, operational and programming measures. Thusly, in every one of the ways, AWS enables organizations to bring down their IT costs.

    For More Info:- AWS Institute in Gurgaon

    ReplyDelete
  8. excellant information please keep sharing such useful information.
    PACKERS AND MOVERS

    ReplyDelete
  9. Finding the best Help with Medical Assignment is not easy unless one is keen to establish a professional medical assignment help & medical homework help online.

    ReplyDelete
  10. Superb topic Resource Management you share.Thanks for taking the time to discuss this. I feel about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.Get best Mobile App Development Dubai you visit here for more info.

    ReplyDelete
  11. Hello! I think these information Will be helpful for you.

    openergroup.com
    kopithecat25.wixsite.com/style1982
    Thank you!

    ReplyDelete
  12. Such a great post you share with us, I really appreciate your work and content idea and I have found here lots of knowledgeable information this website perfect for my need. for More Information Please Visit: Outsource SEO Link Building Services

    ReplyDelete
  13. what great info. It is truly amazing. I have not read this type of post to date. Thank you very much for sharing this information.
    Is the print job being interrupted by Epson Error Code 0XF1? Do not waste outside. You can find a solution. In this report, we have provided a list of solutions for Epson Printer Error 0XF1. To remove this Epson 0XF1 printer error code, you may visit our website.

    ReplyDelete
  14. Hey, what a blog you write,is such wonderful wrinting i never read this ind of stuff.
    Lets talk about one of the best senior care sevices all over Colorado or sarrounding areas. So i rescomand you the Gardens Care Homes Company who is top leading organization in the country for assisted living indutry. Dont worry about the prices, our cheap rates always suits your pocket.
    Arvada Assisted Living
    Lakewood Assisted Living
    Castle Rock Assisted Living
    Aurora Assisted Living
    Federal Heights Assisted Living
    Denver Assisted Living
    Colorado Assisted Living

    ReplyDelete
  15. Some facts I agree to your points but some I don't. Yes, I want to appreciate your hardwork for sharing this information but at my part I have to research more. Though there are some interesting view angle I could find in your remark. Thanks for sharing.
    hire wordpress developer india
    php developers
    outsource digital marketing services

    ReplyDelete
  16. I have just come across your post and I believe this is exactly what I am looking for. I want an economics assignment help from a tutor who can guarantee me a top grade. Do you charge per page or does it depend on the
    bulk of the economics homework help being completed? More to that if the work is not good enough do you offer free corrections.

    ReplyDelete
  17. Matlab Assignment Help helped me to complete my seventh Matlab assignment, which was also the best-performed! It scored 92/100, which I've never scored before on any other assignment/exam in my lifetime. Otherwise, their service was as quick as usual. The delivery was also on time. I'm now requesting to use this same programmer multiple times. He seems the best in Image Processing tasks. Meanwhile, I'll ask for more Matlab Homework Help soon.

    ReplyDelete
  18. I am looking for a Statistics Assignment Help expert for Statistics Homework Help. I have struggled enough with statistics and therefore I just can't do it anymore on my own. . I have come across your post and I think you are the right person to provide me with SPSS homework help. Let me know how much you charge per assignment so that I can hire you today.

    ReplyDelete
  19. Besides my C course, I have a job and family, both of which compete to get my time. I couldn't find sufficient time for the challenging C assignments, and these people came in and saved my skin. I must commend them for the genius Programming Assignment Help. Their C Homework Help tutors did the best job and got me shining grades.

    ReplyDelete
  20. Hi, other than economics assignment help are there other subjects that you cover? I am having several assignments one needs an economics homework help expert and the other one needs a financial expert. If you can guarantee quality work on both then I can hire you to complete them. All I am sure of is that I can hire you for the economics one but the finance one I am not sure.

    ReplyDelete
  21. Me and my classmates took too long to understand Matlab Assignment Help pricing criteria. we're always grateful for unique solutions on their Matlab assignments. Matlab Homework Help experts have the right experience and qualifications to work on any programming student's homework. They help us in our project.

    ReplyDelete
  22. Hey STATA homework help expert, I need to know if you can conduct the Kappa measurement of agreement. This is what is in my assignment. I can only hire someone for statistics assignment help if they are aware of the kappa measurement of agreement. If you can do it, then reply to me with a few lines of what the kappa measure of agreement is. Let me know also how much you charge for statistics homework help in SAS.

    ReplyDelete
  23. The ardent Programming Homework Help tutor that nailed down my project was very passionate. He answered my Python questions with long, self-explanatory solutions that make it easy for any average student to revise. Moreover, he didn't hesitate to answer other questions, too, even though they weren't part of the exam. If all Python Homework Help experts can be like this then they can trend as the best Programming school ever online.

    ReplyDelete
  24. Hello. Please check the task I have just sent and reply as soon as possible. I want an adjustment assignment done within a period of one week. I have worked with an Accounting Homework Help tutor from your team and therefore I know it’s possible to complete it within that period. Let me know the cost so that I can settle it now as your Accounting Assignment Help experts work on it.

    ReplyDelete
  25. That is a huge number of students. Are they from the same country or different countries? I also want your math assignment help. I want to perform in my assignments and since this is what you have been doing for years, I believe you are the right person for me. Let me know how much you charge for your math homework help services.

    ReplyDelete
  26. I don’t have time to look for another expert and therefore I am going to hire you with the hope that I will get quality economics assignment help .Being aneconomics homework help professor I expect that your solutions are first class. All I want to tell you is that if the solutions are not up to the mark I am going to cancel the project.

    ReplyDelete
  27. Hey there, I need an Statistics Homework Help expert to help me understand the topic of piecewise regression. In our lectures, the concept seemed very hard, and I could not understand it completely. I need someone who can explain to me in a simpler way that I can understand the topic. he/she should explain to me which is the best model, the best data before the model and how to fit the model using SPSS. If you can deliver quality work then you would be my official Statistics Assignment Help partner.

    ReplyDelete
  28. Water damage can be a homeowner’s worst nightmare. Not only is your home rendered unlivable for the foreseeable future, but you’ve got a massive water damage cleanup and restoration process to deal with.

    ReplyDelete
  29. The Original Forex Trading System: tradeatf Is The Original Forex Trading System. It Is 100% Automated And Provides An Easy-to-follow Trading System. You Get Access To Real-time Signals, Proven Methods, And A Money-back Guarantee.

    ReplyDelete
  30. Daebak!! this was an extremely good post. Taking the time and actual effort to produce a top notch article… But what can I say… Just check my website and see more 카지노사이트

    ReplyDelete
  31. I had this article saved some time before but my computer crashed. I have since gotten a new one and it took me a while to locate this! 파워볼

    ReplyDelete
  32. Great article, This post helps me a lot Thank you. Anyways I have this site recommendation for you, Just follow the given link here:

    스포츠토토
    토토
    안전놀이터
    토토사이트

    ReplyDelete
  33. I came onto your blog while focusing just slightly submits. Nice strategy for next, I will be bookmarking at once seize your complete rises
    경마사이트

    magosucowep

    ReplyDelete
  34. This is the perfect post.안전놀이터 It helped me a lot. If you have time, I hope you come to my site and share your opinions. Have a nice day.

    ReplyDelete
  35. Thanks for your post! Through your pen I found the problem up interesting! I believe there are many other people who are interested in them just like me! How long does it take to complete this article? I have read through other blogs, but they are cumbersome and confusing.
    happy wheels

    토토사이트
    메이저사이트 목록

    ReplyDelete
  36. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man, Keep it up

    토토사이트
    먹튀검증

    ReplyDelete
  37. I have read your blog and I gathered some new information through your blog. Thanks for sharing the information

    온라인카지노
    카지노

    ReplyDelete
  38. It's perfect time to make a few plans for the future and it is time to be happy. I've learn this submit and if I may just I desire to recommend you some attention-grabbing things or advice. Maybe you could write subsequent articles referring to this article. I want to read even more issues approximately it!



    Also visit my web page :

    카지노사이트추천
    온라인카지노

    ReplyDelete
  39. AximTrade Review Offers A Safe And Secure Platform To Do Forex Trading And CFDs And Our Customer Support Is Ready To Help You 24/7. You Can Easily Sign Up Your Aximtrade Login Account Here.

    ReplyDelete
  40. I do not even know how I ended up here, but I thought this post was good. I do not know who you are but definitely you are going to a famous blogger if you aren’t already ?? Cheers!
    II안전토토사이트

    ReplyDelete
  41. Thanks for sharing this amazing and nice post. Looking for the best dissertation help uk turnout to Assignments Planet for all dissertation services at a cheap price.

    ReplyDelete
  42. That's a really impressive new idea! 메이저놀이터 It touched me a lot. I would love to hear your opinion on my site. Please come to the site I run once and leave a comment. Thank you.


    ReplyDelete
  43. Hello ! I am the one who writes posts on these topics크레이지슬롯 I would like to write an article based on your article. When can I ask for a review?


    ReplyDelete
  44. I accidentally searched and visited your site. I still saw several posts during my visit, but the text was neat and readable. I will quote this post and post it on my blog. Would you like to visit my blog later? keonha cai


    ReplyDelete
  45. I've been looking for photos and articles on this topic over the past few days due to a school assignment, 안전놀이터 and I'm really happy to find a post with the material I was looking for! I bookmark and will come often! Thanks :D


    ReplyDelete
  46. I've learn a few excellent stuff here. Definitely value bookmarking
    for revisiting. I wonder how so much effort you set to create
    such a magnificent informative website. 토토사이트


    ReplyDelete
  47. Youre so right. Im there with you. Your weblog is definitely worth a read if anyone comes throughout it. Im lucky I did because now Ive received a whole new view of this. 메이저사이트

    ReplyDelete
  48. I've been troubled for several days with this topic. 메이저놀이터추천, But by chance looking at your post solved my problem! I will leave my blog, so when would you like to visit it?


    ReplyDelete
  49. In my opinion, the item you posted is perfect for being selected as the best item of the year. You seem to be a genius to combine 먹튀사이트 and . Please think of more new items in the future!

    ReplyDelete
  50. Wow, that’s what I was searching for, what a information! present here at this blog, thanks admin of this web page. 더킹카지노

    ReplyDelete
  51. I've been searching for hours on this topic and finally found your post. 슬롯사이트, I have read your post and I am very impressed. We prefer your opinion and will visit this site frequently to refer to your opinion. When would you like to visit my site?

    ReplyDelete
  52. I am someone who works on the web. Sometimes I like to visit overseas sites sòng bạc when I have time. Among them, I like the site related to , but your site seems to be optimized for too!! Very good

    ReplyDelete
  53. First of all, thank you for your post. 바카라사이트 Your posts are neatly organized with the information I want, so there are plenty of resources to reference. I bookmark this site and will find your posts frequently in the future. Thanks again ^^

    ReplyDelete
  54. Hi there, I simply hopped over in your website by way of StumbleUpon. Now not one thing I’d typically learn, but I favored your emotions none the less. Thank you for making something worth reading. 먹튀검증업체

    ReplyDelete
  55. There are also articles on these topics on my blog and I hope you visit once and have a deep discussion!casino api

    ReplyDelete
  56. Extremely decent blog and articles. I am realy extremely glad to visit your blog. Presently I am discovered which I really need. I check your blog regular and attempt to take in something from your blog. Much obliged to you and sitting tight for your new post.메이저사이트모음

    ReplyDelete
  57. I read a lot of blog posts, but I never heard of a topic like that.
    Get Instant CDR Australia Writing Help and offer the best writing services to engineers who wish instant migration visa approval in Australia. Our team of writers delivers well-written and fully researched CDR Report at reasonable rates.
    Kindly mail us at Contact@CDRAustralia.Org
    Call us at +61-2 9191 7405

    ReplyDelete
  58. You have excelled yourself this time. This is seemingly the best and concise blog.
    Get help for Engineering CDR Report by our experienced writers.
    Visit Our Services links:
    CDR Report Writing Services
    CDR Report Writers
    CDR Report Writing in Australia

    ReplyDelete
  59. First of all, thank you for your post. 온카지노 Your posts are neatly organized with the information I want, so there are plenty of resources to reference. I bookmark this site and will find your posts frequently in the future. Thanks again ^^


    ReplyDelete
  60. The assignment submission period was over and I was nervous, 우리카지노 and I am very happy to see your post just in time and it was a great help. Thank you ! Leave your blog address below. Please visit me anytime.

    ReplyDelete
  61. It's the same topic , but I was quite surprised to see the opinions I didn't think of. My blog also has articles on these topics, so I look forward to your visit.baccarat

    ReplyDelete
  62. Your article was very impressive to me. It was unexpected information,but after reading it like this 온카지노, I found it very interesting.

    ReplyDelete
  63. When I read an article on this topic, 메가슬롯 the first thought was profound and difficult, and I wondered if others could understand.. My site has a discussion board for articles and photos similar to this topic. Could you please visit me when you have time to discuss this topic?

    ReplyDelete
  64. I am contemplating this topic. I think you can solve my problems. My site is at "온라인카지노". I hope you can help me.

    ReplyDelete
  65. Hashlob paved the way as one of the notable, trustworthy and diligent company with a
    diversied array ofsatised customers
    social media marketing in pakistan

    ReplyDelete
  66. I am very impressed with your writing카지노게임 I couldn't think of this, but it's amazing! I wrote several posts similar to this one, but please come and see!

    ReplyDelete
  67. While looking for articles on these topics, I came across this article on the site here. As I read your article, I felt like an expert in this field. I have several articles on these topics posted on my site. Could you please visit my homepage? 메이저놀이터순위

    ReplyDelete
  68. Everything is very open with a really clear explanation of the challenges. It was definitely informative. Your site is useful. Thank you for sharing. Feel free to visit my website; 카지노사이트링크

    ReplyDelete
  69. Hi! I just would like to give you a huge thumbs up for the great info you have got right here on this post. I'll be coming back to your site for more soon. Feel free to visit my website; 온라인카지노사이트넷

    ReplyDelete
  70. Such a valuable post. I like it very much and I like your choice words also. I am waiting for your next valuable post. Feel free to visit my website; 배트맨토토프로

    ReplyDelete
  71. I have read your article; it is very informative and helpful for me. I admire the valuable information you offer in your articles. Thanks for posting it. Feel free to visit my website; 카지노사이트위키

    ReplyDelete
  72. Get the latest Hofv Stocktwits price with our real-time quote and chart. Check the performance of Hofv Stocktwits with historical data and read summary analysis on this stock in Our Servlogin Webpage.

    ReplyDelete
  73. Forex Trading Reviews Is An Online Portal That Brings You The Best Brokers & Strategies For Forex. Along With Our Broker Review And Login Details, We Offer Blog Post Writing, Content Marketing Services, Press Release And Much More! Visit Now & Get Started!

    ReplyDelete
  74. I hope you're doing really well and enjoying the day ahead. We are a Forex Trading agency focusing on the Stock information Brokers Review And Login Details. Get Latest information On Uavs Stocktwits . Our goal is to provide you with comprehensive data and analysis on this stock in order to help you determine if it is a good fit for your portfolio.

    ReplyDelete
  75. Crypto Seo Services You Can Use Link Building Services To Increase The Visibility Of Your Website. The Service Is An Affordable And Long-term Solution To Help Your Site Rank Higher In Search Engine Results. Link Building Services Use A Variety Of Strategies To Help Your Site Rank Higher. One Such Strategy Is To Create Backlinks To Your Site. The Link Building Service Will Create And Place Links On Websites That Are Relevant To Your Business And Industry. This Helps Your Site Rank Higher In Search Engine Results And Increases The Likelihood Of Potential Customers Finding Your Site.

    ReplyDelete
  76. I'm looking for a lot of data on this topic. The article I've been looking for in the meantime is the perfect article. Please visit my site for more complete articles with him! 메이저검증

    ReplyDelete
  77. Are Forex Trading And Forex Brokers Confusing? Check Out Over 25 Highly Rated, Tried-and-tested Forex Brokers. We'll Help You Find The Right One For Your Needs. At Logijn07 Our Mission Is To Help You Find The Right Broker So You Can Get On With What Really Matters, Trading!

    ReplyDelete
  78. Logijn07 is a website that aims to help traders and investors find the right brokers and strategies for online trading. Along with broker review and login details at Logijn07 we offer guest posts, content marketing services, building services and investment guides. On Logijn07 you will also find the latest press releases, industry news and price quotes that might effect investment decision making.

    ReplyDelete
  79. The Truff Stock Overview, keep track of all your favorite stocks in real time across multiple screens, tailored to only show information you need and see as it happens.

    ReplyDelete
  80. The Bfarf Stock Overview, keep track of all your favorite stocks in real time across multiple screens, tailored to only show information you need and see as it happens.

    ReplyDelete
  81. Track your stocks in all major markets instantly with our unique live stock overview. See Ecdp stock live prices changes as they occur and view change details, including volume and share changes.

    ReplyDelete
  82. The most comprehensive guide to Forex Brokers and How To Start Forex Brokerage , showing you what you can save or earn by choosing the best broker for your needs – from Highest to Lowest Charges. Useful for both beginner and seasoned trader.

    ReplyDelete
  83. Stay In Touch With Clii Stock Price? Every Move With Our Live, Real Time Stock Market Overview. Here You Will Always Be One Of The First To Know When Clii Stock Price Is Increasing Or Decreasing. Our Live Information Is Delivered Right To Your Browser, Giving You All The Data You Need To Make An Informed Trading Decision. With This Information At Your Fingertips You Will Be Able To Make Quick And Educated Decisions While Keeping Track Of All The Latest Clii Stock Price Changes!

    ReplyDelete
  84. Dbd Forums Real-Time Overview Of A Stock, Including Recent And Historical Price Charts, News, Events, Analyst Rating Changes And Other Key Stock Information.

    ReplyDelete
  85. White Label Broker is a comprehensive guide to deciding on the best Forex Brokers, Trading Costs and Fees. Find out which brokers offer the best value for money?

    ReplyDelete
  86. Are you a shareholder or investor in Ipof Stocktwits? Our Live, Real Time Stock Market Overview keeps you apprised of all the Ipof Stocktwits happenings during the day. Get comprehensive data on stocks such as price and volume. Our Live, Real Time Stock Market Overview lets you keep up to date on the stock market with no hassles.

    ReplyDelete
  87. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    artificial intelligence internship | best final year projects for cse | internship certificate online | internship for mba finance students | internship meaning in tamil

    ReplyDelete
  88. Are you looking to make money by trading currencies on the forex market? Do you want to find the best forex brokers with the lowest costs and fees? Our free guide provides all the information you need to start trading, selecting a reputable and safe broker, and understanding your White Label Forex Broker Meaning .

    ReplyDelete
  89. As a new forex trader you need to know what the best brokers offers and their White Label Forex Broker Meaning before you begin trading and investing, Use this guide to learn about trading costs, leverage and leverage limits, spreads and execution times, custodial services and market maker spreads .

    ReplyDelete
  90. White Label Partnership Forex is an online forex trading course designed for beginner traders to learn basic and advanced techniques of the forex market. The course is divided into 10 modules that cover topics such as risk management and position sizing strategies along with highly profitable algorithmic trading strategies.

    ReplyDelete
  91. Visit
    www.hp.com/go/wirelessprinting and open the door to the world of HP smart printing solutions.
    Visit ij.start.canon | ij.start canon and find out the best way to download Canon printer drivers.
    Once you are done with the driver setup via
    canon.com/ijsetup , you will have to insert a pile of pages into the printer tray for printing the documents.

    ReplyDelete
  92. White Label Forex Meaning is an effective and affordable way to learn from existing expert traders, so you can use their techniques to start trading profitably for yourself.

    ReplyDelete
  93. 바카라사이트
    This is an awesome article, Given such an extraordinary measure of data in it, These sort of articles keeps the customers excitement for the site, and keep sharing more ... favorable circumstances.

    ReplyDelete
  94. 토토
    Your way of telling everything in this article is genuinely pleasant, all can easily understand it, Thanks a lot

    ReplyDelete
  95. 바카라사이트
    From some point on, I am preparing to build my site while browsing various sites. It is now somewhat completed. If you are interested, please come to play with

    ReplyDelete
  96. At Tradingzy, we can help you learn what is forex seo and how to use it. We're a digital marketing agency focusing on SEO for the Forex Brokerage company. We're familiar with Google's updates and how to work with them effectively.

    ReplyDelete
  97. Forex Seo is an effective and affordable way to learn from existing expert traders, so you can use their techniques to start trading profitably for yourself.

    ReplyDelete
  98. Tradingzy is the best and most reliable Seo company And offers a wide variety "Forex Seo Services. We have a team of professional traders and marketing experts who have extensive experience in the field of digital marketing and forex trading, so we can offer you the best services.

    ReplyDelete
  99. Do you believe in long term investement . One of the option of doing investement is by investing in Crypto currencies. You can invest in Fudxcoin company that deals in the selling and purchasing of Crypto Currency. It is a reliable company. One need not doubt in investing in it as i have also bought crypto currency from it and feeling very satisfied with their services.
    crypto currency blockchain technology

    ReplyDelete
  100. From one day, I noticed that many people post a lot of articles related to 온라인슬롯 . Among them, I think your article is the best among them!!I

    ReplyDelete
  101. The ij.start.cannon setup process for every Canon model is almost similar, however the download through http //ij.start.cannon and installation process may differ.

    ReplyDelete
  102. You can easily set up your Canon printer through drivers from wireless connection, USB, and a few components. The setup process for every Canon model is almost similar, however the download through https //ij.start.cannon and installation process may differ.

    ReplyDelete
  103. Ij start canon and find out the best way to download Canon printer drivers.
    Ij.start.cannon | Ij.start.canon | canon.com/ijsetup

    ReplyDelete
  104. Does anyone one wants to enjoy the delcious food? that to from their favourite resturants. Here comes a company that is offering a wide variety of services like home food delivery, medicine delivery,grocery products and dairy products at the customers doorsteps. It is just a matter of a click, on their app (fudx app) and in no time, you will get your ordered products at your doorsteps.
    food delivery medicine delivery dairy products grocery items

    ReplyDelete
  105. Amazing website, Love it. Great work done. Nice website. Love it. This is really nice.
    locast.org/activate
    hbomax/tvsignin
    disneyplus.com/begin
    showtimeanytime.com/activate

    ReplyDelete
  106. I am actually happy to read this website posts which carries plenty of helpful data, thanks for providing these kinds Feel free to visit my website;
    한국야동

    ReplyDelete
  107. I just found this blog and have high hopes for it to continue. Keep up the great work, its hard to find good ones. I have added to my favorites. Thank You Feel free to visit my website;
    국산야동

    ReplyDelete
  108. Maybe you can write next articles referring to thisarticle. I want to read even more things about it! Feel free to visit my website;
    일본야동

    ReplyDelete
  109. I’ve been trying for a while but I never seem to get there! Thank you Also visit my site: Feel free to visit my website; 일본야동

    ReplyDelete
  110. This is a great article and great read for me. It's my first visit to your blog, and I have found it so useful and informative especially this article
    Yellowstone Dutton Ranch Merch

    ReplyDelete
  111. Howdy! Do you know if they make any plugins to assist with SEO? I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good results. If you know of any please share. Cheers! 안전토토사이트

    ReplyDelete
  112. Our list of the Best MT5 Forex Brokers In Malaysia is compiled solely of trusted, regulated and reputable forex brokers. We have selected well-known brokers that offer great customer service, low fees, an easy-to-use online platform, and education for new investors.

    ReplyDelete
  113. Top Forex Brokers In Malaysia Is A Great Trading Platform, But Is It The Best? Read This Review To See If It Is The Right Option For You.

    ReplyDelete
  114. AFM Logistics Pvt Ltd is an international freight forwarding and customs clearing company established in Delhi. The company was constituted in 2012 and is indulged in providing complete logistics solution. The company has its own setup and wide network of agents throughout the world. Best Custom Clearing Company in Delhi . They are the best air cargo and ocean freight forwarding company in Delhi, India. AFM Logistics Pvt Ltd has been working as Import and Export Agent in India since 2012. They have been providing personal baggage shipping services in India for a very long time.

    ReplyDelete
  115. if anyone wants to leads a healthy lifestyle, he or she can do so by following a weight loss tips, or by following a healthy lifestyle laid down by the nutritionist. Here comes one clinic which can make you fit and healthy by making you follow their diet plans and certain remedies laid down by her . Dr. Namita Nadar is the Best Dietitian and Nutritionist in Noida and Delhi NCR. The clients are very happy with her as she has improved their quality of living with her good diet plans according to the customers requirements. She is the Best Dietitian and nutritionist in Noida NCRif anyone wants to avail her services, one can call on their number or contact through their website or mail on their gmail id https://namitadietclinicnoida.com

    drnamitanadardietclinic@gmail.com
    +919540364364

    ReplyDelete
  116. You made some good points there. I did a Google search about the topic and found most people will believe your blog. 먹튀검증

    ReplyDelete
  117. Operating from New Delhi, Delhi, India, BK IDNSUTRIES came into existence in the year 1963. We are known in the national and international market as a trustworthy manufacturer and we deal in all type of single & double facers corrugated rollers and industrial gears. Our speciality is Best UV Shaped Corrugating Rolls. Our products can meet the demand of different clients with a large quantity of models and complete specifications.

    ReplyDelete
  118. Local authorities say the south-eastern city of Mariupol remains under heavy assault by Russian forces, but around 30,000 people have managed to flee -> 카지노 커뮤니티

    ReplyDelete
  119. The List Of Credit Cards Forex Brokers In Malaysia Is A Perfect Place To Discover The News Of Great Brokers In The Industry. Along With Our Review Of Popular Broker Sites, You Can Also Find Useful Articles That Will Help You Learn More About Trading And Try Out Some Strategies On Your Own.

    ReplyDelete
  120. You’ve written nice post, I am gonna bookmark this page, thanks for info. I actually appreciate your own position and I will be sure to come back here. 카지노사이트

    ReplyDelete
  121. Your post is very helpful and information is reliable. I am satisfied with your post. Thank you so much for sharing this wonderful post. If you have any assignment requirement then you are at the right place. 메이저사이트

    ReplyDelete
  122. Want to know today's latest news, breaking news headlines, and Top news, Discover national and international news on the economy, politics, and defense on a single platform. Star X News24 is the best platform for the latest news and articles. Here, You can read the latest articles and news updates easily. To know more details, visit our website.

    ReplyDelete