Biological, clinical, and ethical advances of placebo effects

Biological, clinical, and ethical advances of placebo eff ects
Damien G Finniss, Ted J Kaptchuk, Franklin Miller, Fabrizio Benedetti Lancet 2010; 375: 686–95
For many years, placebos have been defi ned by their inert content and their use as controls in clinical trials and
University of Sydney Pain
treatments in clinical practice. Recent research shows that placebo eff ects are genuine psychobiological events
Management and Research
attributable to the overall therapeutic context, and that these eff ects can be robust in both laboratory and clinical
Institute, Royal North Shore
settings. There is also evidence that placebo eff ects can exist in clinical practice, even if no placebo is given.
Hospital, St Leonards, NSW,
Further promotion and integration of laboratory and clinical research will allow advances in the ethical use of
Australia
(D G Finniss MSc [Med]); Osher
placebo mechanisms that are inherent in routine clinical care, and encourage the use of treatments that stimulate
Research Center, Harvard
placebo eff ects.
Medical School, Boston, MA,
USA (T J Kaptchuk); Department
Introduction
of Bioethics, National
Institutes of Health, Bethesda,
The notion of something called “placebo” started with laboratory and clinical settings could lead to a MD, USA (F Miller PhD); and
St Jerome’s mistranslation of the fi rst word of the ninth reconsideration of placebo eff ects with implications for Department of Neuroscience,
line of Psalm 116, when instead of translating the clinical practice.
University of Turin Medical
Hebrew “I will walk before the Lord”, he wrote “Placebo School, and National Institute
of Neuroscience, Turin, Italy
Domino in regione vivorum” (“I will please the Lord in Conceptual background
the land of the living”). By the 13th century, when hired The association of placebo eff ects with RCTs has caused mourners waited for Vespers for the Dead to begin, confusion because the response in the placebo group is they often chanted the ninth line, and so were called not necessarily a genuine psychosocial response to the “placebos” to describe their fake behaviour.1 Later, in simulation of treatment. In fact, the reported response The Canterbury Tales, Chaucer named his sycophantic, to placebo in RCTs might refl ect the natural course of fl attering courtier Placebo. The introduction of placebo disease, fl uctuations in symptoms, regression to the controls, which entailed the administration of fake mean, response bias with respect to patient reporting of dfi [email protected]
procedures to separate the eff ects of imagination from subjective symptoms, or other concurrent treatments. reality, began in the 16th century with progressive Furthermore, a traditional focus on the inert content of Catholic eff orts to discredit right-wing exorcisms.2 Individuals “possessed” by the devil were given false understanding placebo eff ects,7,8 not to mention holy objects and if they reacted with violent applying them in clinical research and practice.9contortions—as if they were genuine relics of the holy Much of the controversy surrounding placebo eff ects cross or consecrated wafers—it was concluded that relates to how they are considered and then defi ned. their possession was in their imagination. This idea of placebo controls was then used in medical experiments, beginning with the Franklin commission’s debunking Search strategy and selection criteria
of the psychic force of mesmerism or animal magnetism We searched the Cochrane Library (2001–09), Medline (1902–2009), PreMedline, and Embase (1966–2009) The use of the word placebo in a medical context to databases for reports published in English using the search describe innocuous treatments to make a patient terms “placebo”, “placebo eff ect”, “placebo response”, comfortable dates from at least the end of the “nocebo”, “context eff ect”, “patient-therapist interaction”, 18th century.4 The earlier, unsavoury connections “expectation”, and “conditioning”. We mainly selected undoubtedly led to the tainted reputation of reports published in the past 10 years, but did not exclude placebos and placebo eff ects that persisted until very frequently referenced and highly regarded older recently.1 Mainstream interest in placebo eff ects only publications, especially those that were pertinent to the began with the widespread adoption of the randomised history and understanding of placebo eff ects. We also controlled trial (RCT) after World War II, when it was searched the reference lists of articles identifi ed by this times search strategy, particularly the reference lists of systematic dramatically—in placebo control groups.5 Soon after, in reviews and meta-analyses, and selected those we judged his famous proto-meta-analysis, Henry Beecher claimed relevant, including review articles and book chapters. that about 35% of patients responded positively to Reports were included if they studied or discussed the placebo treatment.6 Beecher, however, encouraged an history, ethics, and mechanisms of placebo use and placebo infl ated notion of the “powerful placebo” because he eff ects both in experimental and clinical settings. In the failed to distinguish the placebo response from other case of mechanistic and clinical trials, trials were only confounding factors. Since this time, there has been included if they were controlled; however, rare exceptions increasing interest in investigating placebo eff ects by were made for older and relevant articles in which a control rigorous research methods, especially in the past 10 years. In this Review, we assess whether advances in www.thelancet.com Vol 375 February 20, 2010
Generally, a placebo is seen as an inert substance or procedure and the placebo eff ect (or response) is Psychosocial context surrounding
Response
something that follows administration of a placebo. the patient
The paradox here is that if something is inert, it is by defi nition unable to elicit an eff ect.7,8 This defi nition can be further confused with terms such as active,10 true, Individual patient and clinician factors
eg, patient’s and clinician’s beliefs,
and perceived placebos,11 which are all attempts to better understand placebo eff ects, and other terms such as context eff ects12,13 and meaning responses,7 which have shifted the focus from the use of the word placebo. Interacting with
Nevertheless, the placebo terminology, despite its defects, is too engrained in the scientifi c literature to Interaction between the patient,
replace it at this time, especially in the absence of a clinician, and treatment environment
eg, factors constituting the clinician– To resolve these confusions and better understand placebo eff ects in clinical trials and practice, it is necessary to reconsider placebos and placebo eff ects, shifting the focus from the inert content of a placebo or sham procedure to what the placebo intervention— consisting of a simulated treatment and the surrounding
clinical context—is actually doing to the patient.
Accumulated evidence suggests that the placebo eff ect Figure 1: Contribution of the psychosocial context surrounding the patient (or placebo component of a given
is a genuine psychobiological event attributable to the treatment) to the overall response
overall therapeutic context.9,14 This psychosocial context
can consist of individual patient and clinician factors, anxiety reduction, and meaning.9,39 Although there is a
and the interaction between the patient, clinician, and growing amount of research into these mechanisms,
treatment environment. The treatment environment two principal mechanisms are well supported.
represents the many factors associated with a treatment
The fi rst mechanism involves expectancy: patients context (such as the specifi c nature of the treatment given placebo have expectations of future responses.40 and the way it is delivered) and the patient–clinician Many experiments have used simple verbal cues as relationship, which is a term that encompasses several modulators of expectations.17,33,41 For example, a factors that constitute the therapeutic interaction participant receiving experimentally induced pain is (fi gure 1).12 The placebo intervention is designed to given a topical placebo cream in the context of two simulate a therapeutic context such that the eff ect of diff erent cues: the fi rst that the cream is inert and will the intervention (placebo eff ect) is attributable to the have no eff ect and the second, that the cream is a way in which this context aff ects the patient’s brain, powerful pain killer.41 Such verbal cues have been shown body, and behaviour.9 When an active treatment is to manipulate patients’ expectations and mediate given, the overall response is the result of the treatment ects—eg, placebo analgesic eff ects in itself and the context in which it is given. Such a concept experimental33 and clinical pain,42 placebo-induced allows for progression in understanding of the many changes in motor performance in Parkinson’s factors that make up the psychosocial context disease,23,43 changes in emotions,28 and brain responses surrounding a patient and how these factors, and the in patients with drug addiction.30 Furthermore, the mechanisms by which they operate, can be enhanced in presence of a conditioning protocol to increase expectations results in larger analgesic responses to placebo, showing that expectation can both mediate and Mechanisms of placebo eff ects
modulate placebo eff ects,17,44,45 as well as interact with Some of the mechanisms that underlie placebo eff ects other constructs such as desire and emotion.9,42are summarised in the table, showing that there is not A second mechanism underlying placebo eff ects one placebo eff ect, but many.14–16 These mechanisms can involves classical conditioning.46 Repeated associations be broadly discussed from psychological and neuro- between a neutral stimulus and an active drug (unconditioned stimulus) can result in the ability of the neutral stimulus by itself to elicit a response characteristic Psychological mechanisms
of the unconditioned stimulus. Classical conditioning From a psychological viewpoint, there are many mecha- mechanisms have been shown in both animal34,47,48 and nisms that contribute to placebo eff ects. These human studies,35,36,44,45 although it has been diffi mechanisms include expectations, conditioning, exclude any cognitive component (such as expectation) learning, memory, motivation, somatic focus, reward, in human beings.49,50 Despite this issue, conditioning www.thelancet.com Vol 375 February 20, 2010
Neurobiological mechanisms
Mechanisms
Activation of endogenous opioids and dopamine (placebo); activation of neurobiological viewpoint further emphasises that cholecystokinin and deactivation of dopamine (nocebo)17–22 there are several placebo eff ects. Placebo eff ects can Activation of dopamine in the striatum and changes in activity of neurons in occur in diff erent physiological systems in healthy volunteers and in patients with many diff erent clinical Changes of electrical and metabolic activity in diff erent brain regions (eg, ventral striatum)26,27 Changes in activity of the anterior cingulated and orbitofrontal cortices; Most research into the neurobiology of placebo genetic variants of serotonin transporter and tryptophan hydroxylase 228,29 responsiveness has addressed placebo analgesia; Changes of metabolic activity in diff erent brain regions30 accordingly, the neurobiology of placebo eff ects is Change of neuronal excitability in limbic regions31 usually considered in terms of opioid and non-opioid mechanisms.54,55 Several studies have shown that Reduction of β-adrenergic activity of heart32 placebo eff ects can be completely18,19,56 or partly reversed57 Conditioning of opioid receptors in the respiratory centres33 by the opioid antagonist naloxone, supporting the Conditioning of some immune mediators (eg, interleukin 2, interferon γ, involvement of endogenous opioids in some placebo analgesic eff ects.58 Furthermore, placebo analgesic Conditioning of some hormones (eg, growth hormone, cortisol)36 eff ects are likely to be inhibited by the peptide Activation of endogenous opioids and increased muscle work37,38 cholecystokinin,19 since such eff ects are potentiated in Prefrontal executive control and functional connectivity of prefrontal areas21 patients treated with cholecystokinin antagonist.59,60 Table: Mechanisms for placebo eff ects in medical conditions and physiological systems
Several studies have shown that placebo eff ects can occur at specifi c body regions.33,41,61 This body-region specifi city is reversed by naloxone,33 suggesting that mechanisms in human beings are substantiated by the analgesic responses to placebo involve highly specifi c fact that placebo eff ects are higher in magnitude after a endogenous opioid release, rather than a more conditioning protocol (even if an expectation mechanism generalised opioid release (such as increased opioid is present).17 Additionally, conditioning mechanisms concentration in the cerebrospinal fl uid).62 These results mediate placebo-induced changes in unconscious have been substantiated and extended by brain imaging physiological processes such as hormone secretion36 and techniques such as PET63,64 and functional MRI.65–67 In one PET study, brain changes in response to placebo The interaction between expectation and conditioning were reported to be similar to changes seen after mechanisms remains an area for further research, treatment with opioid drug.68 Opioid-mediated placebo which might be particularly relevant to exploring the responses also extend beyond pain pathways. Some clinical implications of these mechanisms. Although studies have shown that placebo-induced respiratory classical conditioning, manifesting an automatic depression (a conditioned placebo side-eff ect)69 and unconscious mechanism, exists in human beings, it decreased heart rate and β-adrenergic activity32 can be can also be regarded as a complex process consisting of cognitive components and derived from previous Many placebo eff ects are mediated by non-opioid experience of either positive or negative therapeutic mechanisms, such as the release of diff erent neuro-outcomes.51 Accordingly, conditioning and expectation transmitters and neuromodulators. In one study, the are certainly entangled in the occurrence of placebo placebo response in participants who had previous eff ects in clinical practice. The most reasonable conditioning with an opioid drug was reversed by interpretation of recent publications is that conditioning naloxone; however, there was no reversal in those who had follows expectation and is dependent on the success of conditioning with a non-opioid drug.17 Therefore, the fi rst encounter. This notion leads to the completely diff erent placebo mechanisms can be produced possibility that the fi rst encounter is crucial for the depending on the drug used in the conditioning protocol.
development of subsequent robust placebo responses: Although other medical disorders have been the higher the expectation, the greater the placebo eff ect, investigated from a neurobiological perspective, the and potentially the greater the conditioning placebo mechanisms in these conditions are little eff ects associated with future drug intake.
understood compared with those for pain and analgesia. In addition to classical conditioning, other learning For example, in patients with Parkinson’s disease, processes such as past experiences and social administration of placebo led to dopamine release in observation mediate placebo eff ects.52 For example, the striatum23,70 and resulted in changes in basal ganglia participants who observed a demonstrator simulating and thalamic neuron fi ring.24,25 Studies have also shown responsiveness to a therapy had placebo analgesic changes in metabolic activity in the brain after responses that were similar in magnitude to those in administration of placebo in patients with depression26 patients who received a classical conditioning and after manipulation of expectations in patients with procedure.53 www.thelancet.com Vol 375 February 20, 2010
Psychosocial context
Figure 2: Receptor pathways activated by both psychosocial context and drugs
Social stimuli around the treatment might activate, through expectation or conditioning mechanisms, several receptor pathways in diff erent diseases and
therapeutic interventions (the involvement of serotonin [5-hydroxytryptamine; 5HT] receptors in hormonal responses and depression is not defi nitive). These
receptors are the same to which diff erent drugs bind, suggesting that psychosocial factors are capable of modulating the action of drugs. This interference has
implications for our understanding of drug action: when a drug is prescribed, the very act of giving it to a patient (ie, the psychosocial context) might aff ect the
system and change the response to the drug. Reproduced with permission from reference 39. IFNγ=interferon γ. IL2=interleukin 2. CCK=cholecystokinin.
Less research has been concerned with the nocebo representative of normal clinical practice than a clinical eff ect, an occurrence that is opposite to the placebo trial setting. It is therefore important to bridge this gap eff ect. The reason for the paucity of data is mainly by looking at placebo research from basic science, because of ethical limitations, since nocebo clinical trial, and ethical perspectives in an attempt to administration involves the induction of negative better understand how placebo eff ects operate in the expectations. Cholecystokinin has a key role in nocebo clinical setting.
hyperalgesia, which occurs through anticipatory anxiety A single-blind RCT in 262 patients with irritable mechanisms.20,21,37 De activation of dopamine release has bowel syndrome investigated whether placebo eff ects also been found in the nucleus accumbens during can be disaggregated into two main components nocebo hyperalgesia,22 which suggests the involvement (placebo ritual alone and placebo ritual plus supportive of diff transmitters. Furthermore, a patient–clinician relationship) and then progressively neuroimaging study has shown that nocebo aff ects combined to produce clinically signifi cant improvements brain activation diff erently from placebo, including in compared with no treatment.77 The placebo ritual the hippocampus and regions involved with antici- consisted of a validated placebo acupuncture device, which was used in both treatment groups.78 Instead of penetrating the skin, the needle retracts into the needle Implications for clinical practice
handle. The supportive patient–clinician relationship, Understanding how placebo eff ects work clinically in used only in one group, was prospectively scripted and relevant patient populations over time has not kept pace consisted of attention, warmth, confi dence, and with the recent research into mechanisms of placebo thoughtful silence. At the 3-week outcome, adequate eff ects, which has mainly involved laboratory relief on a validated measure for irritable bowel experiments done over short durations with healthy syndrome was reported by 62% of participants in the participants. In the case of clinical populations, the placebo ritual plus supportive care group, 44% in the study of long-term placebo responsiveness has been placebo ritual alone group, and 28% in the no-treatment limited to RCTs. However, these studies rarely included group (p<0·001). The results were similar with three groups of participants receiving no treatment to control other validated measures for irritable bowel syndrome for natural history and regression to the mean, making used in the study. The eff ect size of 62% adequate relief cult to discern a genuine placebo eff ect. Several was similar to the improvement seen in patients treated meta-analyses have attempted to address the presence with alosetron in RCTs of irritable bowel syndrome.79 and magnitude of placebo eff ects in RCTs, including Outcomes were similar after an additional 3 weeks of some studies in which no-treatment control groups follow-up. In addition to showing that genuine placebo were used. These analyses concluded that placebo eff ects can be statistically and clinically signifi cant over eff ects are small and limited to subjective outcomes time in clinical populations, this trial showed that when placebos are used as a control condition in placebo eff ects can be incrementally added in a manner RCTs.72–74 However, placebo eff ects are much larger in resembling a graded dose escalation of component studies that investigate placebo mechanisms.75,76 This factors. In a separate analysis of the study, patient fi nding is not at all surprising given that the mechanistic extroversion, agreeableness, and openness to experience experiments use controlled manipulations of verbal were found to be associated with placebo responses in instructions and context that might be more the placebo ritual plus supportive care group but not in www.thelancet.com Vol 375 February 20, 2010
(amitriptyline) and the sham acupuncture group was Routine medical practice
Although 30% of people in both placebo groups reported adverse eff ects, the type of eff ects diff ered between groups and mimicked the information provided during the informed consent process.
Some commentators have suggested that alternative therapies with elaborate procedures and distinct environmental cues might have pronounced and clinically signifi cant placebo eff ects.83,84 Recent RCTs of acupuncture, although not designed to study placebo Treatment simulation with placebo
eff ects, have provided results that lend support to this hypothesis. A series of large trials in Germany compared acupuncture done according to traditional Chinese medicine (verum acupuncture), sham acupuncture (superfi cial needling at non-acupuncture points), and either no-treatment or usual clinical care. Conditions studied included migraine,85 tension headaches,86 Hidden dose of active treatment
chronic low back pain,87,88 and osteoarthritis of the knee.89 Generally across the various trials, outcomes did not diff er between verum and sham acupuncture groups; however, participants in both of these groups had substantially greater symptom improvement than did those in the no-treatment and usual clinical care Figure 3: Rationale of the open-hidden study design
control groups.90 Linde and colleagues91 reported that in In routine clinical practice, any treatment has a specifi c and a non-specifi c eff ect. four of these RCTs (n=864), patient’s expectation of The non-specifi c eff ect might come from the knowledge that a treatment is pain relief was the most robust predictor of effi being given. The eff ectiveness of the active treatment can be assessed either by eliminating its specifi c eff ect (placebo study) or by eliminating the non-specifi c acupuncture treatment, irrespective of the group eff ects (hidden treatment). Reproduced from reference 94. assignment to genuine or sham treatment. The eff ect of positive expectation on outcome lasted for 1 year. These the placebo ritual alone group.80 The investigators also results therefore accord with the hypothesis that reported signifi cant diff erences in outcomes between acupuncture works by means of a placebo eff ect. A practitioners. Future integration of such study designs more recent study in 640 patients with chronic low back in RCTs with mechanistic laboratory work will allow for pain showed that participants assigned to 8 weeks of better understanding of these placebo mechanisms and toothpick simulation sham acupuncture plus usual care how they can be augmented in clinical practice.
had clinically meaningful improvements in outcomes Several RCTs have examined whether diff erent compared with those assigned to usual clinical care methods for delivery of placebo produce diff erent alone, and such eff ects also lasted for 1 year.92 This study eff ects.81 The largest such study, in 270 patients with however did not fi nd a correlation between measured chronic arm pain caused by repetitive use, compared a sham device (placebo acupuncture) with an inert oral Some of the clearest evidence supporting the pill.82 At 2 weeks of treatment, patients assigned to involvement of placebo eff ects in clinical care comes placebo pills had greater improvement in ability to from trials with an open-hidden study design (fi gure 3). function (mainly related to less disturbed sleep because In this experimental approach, a treatment is given in a of pain) than did patients assigned to sham acupuncture routine manner (open treatment), in which the (p<0·05); however, pain did not diff er between groups. psychosocial context surrounding treatment admin- At the end of the study (6 weeks), patients assigned to istration is present, and in a hidden manner, in which sham acupuncture had a signifi cant reduction in pain the treatment is given without the patient’s knowledge. compared with those in the placebo pill group In the case of a drug intervention, the open treatment (p<0·001). Depending on the complaint and the length mimics normal clinical care; the clinician injects a drug of time that placebo was received, diff erent placebos in full view of the patient with verbal and contextual had diff erent eff ects. Not all medical rituals are the interactions. For the hidden treatment, the drug is same: placebo pills are better for sleep and sham infused by a computer pump in the absence of the needles are better for pain. Nocebo eff ects also diff ered clinician and the therapeutic context. Patients receiving between treatment delivery groups. Patients in the hidden treatment are aware that at some stage they will placebo pill group were told they might have the receive a drug but they do not experience the expectation adverse eff ects (eg, drowsiness) of a medication component or other contextual factors surrounding the www.thelancet.com Vol 375 February 20, 2010
treatment. Because the hidden administration removes Placebo mechanisms can interact with drug treatments, the psychosocial context of treatment, the placebo even if no placebo is given, since every treatment is component is defi ned as the diff erence in outcome given in a therapeutic context that has potential to between open and hidden treatments, although no activate and modulate placebo mechanisms, many of placebo is given.94,95 which can act on similar biochemical pathways to the The open-hidden study design has been used in actual drug (fi gure 2).
several clinical settings. Hidden treatment with widely A short-term experimental trial done in 2001 has used painkillers (morphine, buprenorphine, tramadol, advanced our understanding of the clinical implications ketorolac, metamizol) has been shown to be markedly of modulating placebo eff ects in routine clinical care. less eff ective in reducing pain than has open In this trial, which assessed postoperative pain over treatment.94,96,97 This fi nding was seen in both healthy several days, patients were given intravenous saline volunteers receiving experimentally induced pain (pain (placebo) as a background infusion in addition to ratings were higher in the hidden treatment group routine analgesic treatment (buprenorphine on than in the open treatment group) and in patients with request).98 One group of patients was told that the basal postoperative pain (the dose needed to reduce pain by infusion was a rehydrating solution (natural history 50% was much higher in the hidden treatment group control group) and another group was told that it was a than in the open treatment group).96 Similar diff erences powerful painkiller (maximum placebo context). Overall between open and hidden treatment groups have been intake of buprenorphine was monitored throughout the reported after drug treatment in patients with anxiety trial. The clear diff erences in the context (mainly and after deep brain stimulation in patients with expectation of benefi t) of the basal infusion resulted in Parkinson’s disease.31,97 Slightly diff erent methods have substantial diff erences in drug intake. The group who been used in patients with drug addiction; the absence believed the solution was assisting in analgesia took of an expectation component in patients given 33% less buprenorphine for the same pain control than stimulant drug treatment resulted in reduced regional did those in the natural history control group, showing brain glucose metabolism and verbal reports of an important clinical eff ect and the potential for use of effi cacy.30 Thus, the overall outcome of a treatment placebo eff ects in conjunction with an active treatment combines the specifi c pharmacological or physiological to reduce overall drug intake. A third group were told action of the treatment and the psychosocial context in that “the solution may or may not be a powerful which it is delivered. The psychosocial context painkiller”, representing classic double-blind treatment represents the placebo component, which is based on used in placebo-controlled trials. In this group, patients patient expectations.
took 20% less buprenorphine than did controls.
The open-hidden study design has provided a means Similar modulations in short-term placebo eff ects of exploring the interaction between placebo eff ects and have been reported in more recent studies in patients responses to active treatments. This analysis has not with irritable bowel syndrome.42,99 In these studies, been possible in standard RCTs designed to assess patients were exposed to a painful stimulus (rectal treatment effi cacy, since they only compare the response to placebo with the response to the index intervention anaesthetic and placebo. In one study, patients were without providing an understanding of the interaction told that they “may receive an active or a placebo between the two. For example, fi ndings from a clinical agent”,99 whereas in the second, they were told that “the trial done in 1995 showed that the cholecystokinin agent you have been given is known to signifi cantly antagonist proglumide was more eff ective in reducing reduce pain in some patients”.42 The subtle changes in intensity of postoperative pain than was placebo, which in turn was more eff ective than no treatment.59 responses, with larger placebo responses reported in According to methods of analysis used in classic clinical the second trial, which had more defi nite instructions.
trials, these results would suggest that proglumide is a Clinicians’ expectations also seem to aff ect placebo good analgesic drug that acts on pain pathways, whereas responses. In a small, double-blind trial done in 1985, placebo reduces pain by activating placebo analgesic patients with postoperative dental pain were divided mechanisms (through expectation pathways). However, into two groups and told that they could receive a drug this conclusion is erroneous, since a hidden injection which would increase their pain (naloxone), decrease of proglumide had no analgesic eff ect. If the drug is an their pain (fentanyl), or have no eff ect (placebo).100 By eff ective modulator of pain pathways, such a diff erence contrast, the clinicians were told that in one of the between open and hidden treatment would not be seen. groups, there was no chance of receiving an active In this instance, the drug achieves a response by analgesic drug, and to this extent it was the clinicians interacting with and enhancing placebo mechanisms who were manipulated and not the patients. The (expectation pathways), not by acting on pain pathways, placebo response was substantially lower in the group and therefore is only eff ective when combined with the that clinicians believed would receive no analgesic placebo mechanisms inherent in the clinical encounter. treatment. The double-blind nature of the study www.thelancet.com Vol 375 February 20, 2010
suggests that alterations in clinicians’ beliefs might concerns. To recommend or give a placebo intervention alter the therapeutic context (and placebo eff ect) in deceptively as a treatment with specifi c effi subtle ways, since patients were not aware of the patient’s condition violates informed consent and information given to the clinicians.
threatens the trust that is central to clinical practice.103 Loss of placebo mechanisms can have important Recent data suggest that prescriptions of sugar pills clinical ramifi cations. For example, an open-hidden and saline injections are rare,104,105 but that clinicians study in patients with Alzheimer’s disease showed that often prescribe various active treatments with the main the placebo component (diff erence between open and intent of promoting a placebo response or complying hidden treatments) was correlated with cognitive status with the wishes of the patient. The available evidence and functional connectivity between brain regions.101 suggests that the practice of disclosure to patients Reductions in both cognitive status and functional regarding such placebo treatments is deceptive or at connectivity correlated with reduced placebo least not suffi mechanisms and reduced overall analgesic eff ect, so Can a recommendation for a treatment intended to much so that an increase in dose was needed for the promote the placebo eff ect be made without deception same level of analgesia. This fi nding shows the and also without undermining its therapeutic potential? importance of not only attempting to increase placebo Consider, for example, the case of a clinician who components of treatments, but also of assessing recommends acupuncture treatment for a patient with situations in which loss of placebo mechanisms might chronic low back pain who has not been helped by necessitate an increased therapeutic dose.
standard medical therapy. Aware of the results of the recent acupuncture trials, this clinician thinks that Ethical principles of enhancing placebo eff ects
acupuncture might work by promoting a placebo in clinical care
response. The clinician might provide the following Any ethical assessment of eff orts to promote placebo disclosure to the patient: “I recommend that you try eff ects in clinical practice fi rst requires knowledge as to acupuncture. Several large studies have shown that the clinical relevance and importance of placebo eff ects. traditional acupuncture is not better than fake The evidence reviewed here outlines the potential for acupuncture treatment, but that both of these produce placebo interventions and the therapeutic context to substantially greater symptom improvement in patients promote clinically important symptomatic relief. with chronic low back pain compared with those Nevertheless, more studies of placebo eff ects in specifi c patients who receive no treatment or conventional clinical settings are needed before use of treatments medical therapy. Although the specifi c type of needling with the primary aim of promoting placebo responses does not seem to make any diff erence, it is possible that can be recommended as evidence-based practice.
acupuncture works by a psychological mechanism that A second important ethical consideration relates to promotes self-healing, known as the placebo eff ect”. At whether and how placebo eff ects can be promoted face value, this disclosure seems honest. A patient who without deception. Since placebo eff ects are inherent in received this disclosure and subsequently got better routine clinical care, and the psychosocial context after undergoing acupuncture might nonetheless surrounding the patient (including the patient–clinician develop a false belief about why it worked. This does interaction and the therapeutic procedure) can be not mean, however, that the patient has been deceived enhanced to improve these placebo eff ects, it is ethically acceptable, not to mention clinically relevant, to provide Can it be ethical for clinicians to prescribe inert a supportive clinical encounter that relieves anxiety and placebos with a disclosure that the treatment being promotes positive expectations along with honest given “has been shown to be eff ective by altering pain disclosure of the expected benefi ts of a medically transmission in similar ways to other treatments”? As is indicated treatment. Therefore, routine conscious the case with most studies of the placebo eff ect,102 an attempts to identify and exploit features of the clinical element of deception is involved. In this case, the encounter to augment placebo eff ects represent one element of deception relates to a lack of full disclosure ethical (non-deceptive) means of applying the of the content of the placebo and the complete reason understanding of placebo mechanisms to improve for why it is being given—ie, not only to modulate pain clinical outcomes.
transmission but to do so through a placebo eff ect. Whether it is ethical to recommend a treatment Therefore, as with acupuncture, completely eliminating primarily to produce a placebo eff ect is a more deception would require additional disclosure that the complicated and controversial question. Most studies placebo had no active drug in it and would be working of the placebo eff ect have used deception in the through psychological mechanisms that promote self-administration of inert placebos as a key element of healing. How such disclosure might aff ect placebo experimental design. Whereas the use of deception in responses is unknown, and apart from two small trials research poses its own ethical issues,102 the problem of in patients with various mild psychiatric symptoms (and deception in clinical practice raises even stronger without a no-treatment control group),106,107 no research www.thelancet.com Vol 375 February 20, 2010
has addressed this important question. It is therefore 14 Miller FG, Kaptchuk TJ. The power of context: reconceptualizing important that clinicians who are recommending the placebo eff ect. J R Soc Med 2008; 101: 222–25.
treatments for the primary aim of enhancing placebo 15 Benedetti F. Placebo eff ects: understanding the mechanisms in health and disease. New York: Oxford University Press, 2009.
eff ects are aware of the ethical implications of diff erent 16 Kaptchuk TJ, Shaw J, Kerr CE, et al. “Maybe I made up the whole types of disclosure and the potential for deception. thing”: placebos and patients’ experiences in a randomized Clinically focused research is needed to explore non- controlled trial. Cult Med Psychiatry 2009; 33: 382–411.
17 Amanzio M, Benedetti F. Neuropharmacological dissection of deceptive techniques for prescribing treatments aimed placebo analgesia: expectation-activated opioid systems versus conditioning-activated specifi c subsystems. J Neurosci 1999;
19: 484–94.
Conclusions
18 Levine JD, Gordon NC, Fields HL. The mechanism of placebo analgesia. Lancet 1978; 312: 654–57.
Laboratory evidence supports the existence of several 19 Benedetti F. The opposite eff ects of the opiate antagonist placebo mechanisms and placebo eff ects in both healthy naloxone and the cholecystokinin antagonist proglumide on
placebo analgesia. Pain 1996; 64: 535–43.
volunteers and patients with a variety of medical 20 Benedetti F, Amanzio M, Casadio C, Oliaro A, Maggi G. Blockade conditions. Furthermore, clinically relevant evidence of nocebo hyperalgesia by the cholecystokinin antagonist shows that placebo eff ects can have meaningful proglumide. Pain 1997; 71: 135–40.
therapeutic eff ects, because of their long magnitude 21 Benedetti F, Amanzio M, Vighetti S, et al. The biochemical and neuroendocrine bases of the hyperalgesic nocebo eff ect. J Neurosci and duration, in diff erent patient populations. Although 2006; 26: 12014–22.
substantial progress has been made in understanding 22 Scott DJ, Stohler CS, Egnatuk CM, Wang H, Koeppe RA, placebo eff ects, much laboratory and translational Zubieta JK. Placebo and nocebo eff ects are defi ned by opposite opioid and dopaminergic responses. Arch Gen Psychiatry 2008; clinical trial research remains to be done, with the 65: 220–31.
ultimate aim of harnessing placebo eff ects to improve 23 de la Fuente-Fernandez R, Ruth TJ, Sossi V, Schulzer M, Calne patient care.
DB, Stoessl AJ. Expectation and dopamine release: mechanism of the placebo eff ect in Parkinson’s disease. Science 2001; Contributors
293: 1164–66.
All authors participated in the search of the published work and the 24 Benedetti F, Colloca L, Torre E, et al. Placebo-responsive writing of the report. All authors have seen and approved the fi nal Parkinson patients show decreased activity in single neurons of subthalamic nucleus. Nat Neurosci 2004; 7: 587–88.
25 Benedetti F, Lanotte M, Colloca L, Ducati A, Zibetti M, Lopiano L. Confl icts of interest
Electrophysiological properties of thalamic, subthalamic and We declare that we have no confl icts of interest.
nigral neurons during the anti-parkinsonian placebo response. Acknowledgments
J Physiol 2009; 587: 3869–83.
TJK is supported by a grant from National Center for Complementary 26 Mayberg HS, Silva JA, Brannan SK, et al. The functional Medicine (number K24 AT004095), National Institutes of Health and neuroanatomy of the placebo eff ect. Am J Psychiatry 2002; receives support from Bernard Osher Foundation, San Francisco, CA, 159: 728–37.
USA. FB is supported by grants from Istituto San Paolo and Regione 27 Leuchter AF, Cook IA, Witte EA, Morgan M, Abrams M. Changes Piemonte (Turin, Italy) and Volkswagen Foundation (Hannover, in brain function of depressed subjects during treatment with placebo. Am J Psychiatry 2002; 159: 122–29.
28 Petrovic P, Dietrich T, Fransson P, et al. Placebo in emotional References
processing—induced expectations of anxiety relief activate a Aronson J. Please, please me. BMJ 1999; 318: 716.
generalized modulatory network. Neuron 2005; 46: 957–69.
Kaptchuk TJ, Kerr CE, Zanger A. Placebo controls, exorcisms, and 29 Furmark T, Appel L, Henningsson S, et al. A link between the devil. Lancet 2009; 374: 1234–35.
serotonin-related gene polymorphisms, amygdala activity, and Kaptchuk T. Intentional ignorance: a history of blind assessment placebo-induced relief from social anxiety. J Neurosci 2008; and placebo controls in medicine. Bull Hist Med 1998; 28: 13066–74.
72: 389–433.
30 Volkow ND, Wang G, Ma Y, et al. Expectation enhances the Kerr CE, Milne I, Kaptchuk TJ. William Cullen and a missing regional brain metabolic and the reinforcing eff ects of stimulants mind-body link in the early history of placebos. J R Soc Med 2008; in cocaine abusers. J Neurosci 2003; 23: 11461–68.
101: 89–92.
31 Lanotte M, Lopiano L, Torre E, et al. Expectation enhances Kaptchuk TJ. Powerful placebo: the dark side of the randomised autonomic responses to stimulation of the human subthalamic controlled trial. Lancet 1998; 351: 1722–25.
limbic region. Brain Behav Immun 2005; 19: 500–09.
Beecher HK. The powerful placebo. JAMA 1955; 159: 1602–06.
32 Pollo A, Vighetti S, Rainero I, Benedetti F. Placebo analgesia and Moerman DE, Jonas WB. Deconstructing the placebo eff ect and the heart. Pain 2003; 102: 125–33.
fi nding the meaning response. Ann Intern Med 2002; 136: 471–76.
33 Benedetti F, Arduino C, Amanzio M. Somatotopic activation of Moerman DE. “Placebo” versus “meaning”: the case for a change opioid systems by target-directed expectations of in our use of language. Prevent Treat 2003; 6: 1–5.
analgesia. J Neurosci 1999; 19: 3639–48.
Price DD, Finniss DG, Benedetti F. A comprehensive review of 34 Ader R, Cohen N. Behaviourally conditioned the placebo eff ect: recent advances and current thought. immunosuppression. Psychosom Med 1975; 37: 333–40.
Annu Rev Psychol 2008; 59: 565–90.
35 Goebel MU, Trebst AE, Steiner J, et al. Behavioral conditioning of 10 Stewart-Williams S. The placebo puzzle: putting together the immunosuppression is possible in humans. FASEB J 2002; pieces. Health Psychol 2004; 23: 198–206.
16: 1869–73.
11 Ernst E, Resch KL. Concept of true and perceived placebo eff ects. 36 Benedetti F, Pollo A, Lopiano L, Lanotte M, Vighetti S, Rainero I. BMJ 1995; 311: 551–53.
Conscious expectation and unconscious conditioning in analgesic, 12 Di Blasi Z, Harkness E, Ernst E, Georgiou A, Kleijnen J. Infl uence motor, and hormonal placebo/nocebo responses. J Neurosci 2003; of context eff ects on health outcomes: a systematic review. Lancet 23: 4315–23.
2001; 357: 757–62.
37 Benedetti F, Lanotte M, Lopiano L, Colloca L. When words are 13 Di Blasi Z, Kleijnen J. Context eff ects. Powerful therapies or painful: unraveling the mechanisms of the nocebo eff ect. methodological bias? Eval Health Prof 2003; 26: 166–79.
Neuroscience 2007; 147: 260–71.
www.thelancet.com Vol 375 February 20, 2010
38 Pollo A, Carlino E, Benedetti F. The top-down infl uence of 66 Price DD, Craggs J, Verne GN, Perlstein WM, Robinson ME. ergogenic placebos on muscle work and fatigue. Eur J Neurosci Placebo analgesia is accompanied by large reductions in 2008; 28: 379–88.
pain-related brain activity in irritable bowel syndrome patients. 39 Benedetti F. Mechanisms of placebo and placebo-related eff ects Pain 2007; 127: 63–72.
across diseases and treatments. Annu Rev Pharmacol Toxicol 2008; 67 Kong J, Gollub RL, Rosman IS, et al. Brain activity associated with 48: 33–60.
expectancy-enhanced placebo analgesia as measured by functional 40 Kirsch I. Response expectancy as a determinant of experience and magnetic resonance imaging. J Neurosci 2006; 26: 381–88.
behavior. Am Psychol 1985; 40: 1189–202.
68 Petrovic P, Kalso E, Petersson KM, Ingvar M. Placebo and opioid 41 Price DD, Milling LS, Kirsch I, Duff A, Montgomery GH, analgesia—imaging a shared neuronal network. Science 2002; Nicholls SS. An analysis of factors that contribute to the 295: 1737–40.
magnitude of placebo analgesia in an experimental paradigm. 69 Benedetti F, Amanzio M, Baldi S, Casadio C, Maggi G. Inducing Pain 1999; 83: 147–56.
placebo respiratory depressant responses in humans via opioid 42 Vase L, Robinson ME, Verne GN, Price DD. The contributions of receptors. Euro J Neurosci 1999; 11: 625–31.
suggestion, desire, and expectation to placebo eff ects in irritable 70 de la Fuente-Fernandez R, Stoessl AJ. The placebo eff ect in bowel syndrome patients. An empirical investigation. Pain 2003; Parkinson’s disease. Trends Neurosci 2002; 25: 302–06.
105: 17–25.
71 Kong J, Gollub RL, Polich G, et al. A functional magnetic 43 Pollo A, Torre E, Lopiano L, et al. Expectation modulates the resonance imaging study on the neural mechanisms of response to subthalamic nucleus stimulation in Parkinsonian hyperalgesic nocebo eff ect. J Neurosci 2008; 28: 13354–62.
patients. Neuroreport 2002; 13: 1383–86.
72 Hrobjartsson A, Gotzsche PC. Is the placebo powerless? An 44 Voudouris NJ, Peck CL, Coleman G. Conditioned response models analysis of clinical trials comparing placebo with no treatment. of placebo phenomena: further support. Pain 1989; 38: 109–16.
N Engl J Med 2001; 344: 1594–602.
45 Voudouris NJ, Peck CL, Coleman G. The role of conditioning and 73 Hrobjartsson A, Gotzsche PC. Is the placebo eff ect powerless? verbal expectancy in the placebo response. Pain 1990; 43: 121–28.
Update of a systematic review with 52 new randomized trials 46 Siegel S. Explanatory mechanisms for placebo eff ects: Pavlovian comparing placebo with no treatment. J Intern Med 2004; conditioning. In: Guess HA, Kleinman A, Kusek JW, Engel LW, 256: 91–100.
eds. The science of the placebo: toward an interdisciplinary 74 Hrobjartsson A, Gotzsche PC. Placebo interventions for all research agenda. London: BMJ Books, 2002: 133–57.
clinical conditions. Cochrane Database Syst Rev 2004; 3: CD003974.
47 Herrnstein R. Placebo eff ect in the rat. Science 1962; 138: 677–78.
75 Vase L, Riley JL, Price DD. A comparison of placebo eff ects in 48 Pacheco-Lopez G, Engler H, Niemi MB, et al. Expectations and clinical analgesic trials versus studies of placebo analgesia. Pain associations that heal: immunomodulatory placebo eff ects and its 2002; 99: 443–52.
neurobiology. Brain Behav Immun 2006; 20: 430–46.
76 Hrobjartsson A, Gotzsche PC. Unsubstantiated claims of large 49 Stewart-Williams S, Podd J. The placebo eff ect: dissolving the eff ects of placebo on pain: serious errors in meta-analysis of expectancy versus conditioning debate. Psychol Bull 2004; placebo analgesia mechanism studies. J Clin Epidemiol 2006; 130: 324–40.
59: 336–38.
50 Montgomery GH, Kirsch I. Classical conditioning and the placebo 77 Kaptchuk TJ, Kelley JM, Conboy LA, et al. Components of placebo eff ect. Pain 1997; 72: 107–13.
eff ect: randomised controlled trial in patients with irritable bowel 51 Rescorla RA. Pavlovian conditioning. It’s not what you think it is. syndrome. BMJ 2008; 336: 999–1003.
Am Psychol 1988; 43: 151–60.
78 Steitberger K, Kleinhenz J. Introducing a placebo needle into 52 Colloca L, Benedetti F. How prior experience shapes placebo acupuncture research. Lancet 1998; 352: 364–65.
analgesia. Pain 2006; 124: 126–33.
79 Camilleri M, Northcutt AR, Kong S, Dukes GE, McSorley D, 53 Colloca L, Benedetti F. Placebo analgesia induced by social cacy and safety of alosetron in women with observational learning. Pain 2009; 144: 28–34.
irritable bowel syndrome: a randomised, placebo-controlled trial.
Lancet 2000; 355: 1035–40.
54 Finniss DG, Benedetti F. Mechanisms of the placebo response and their impact on clinical trials and clinical practice. Pain 2005; 80 Kelley JM, Lembo AJ, Ablon JS, et al. Patient and practitioner 114: 3–6.
infl uences on the placebo eff ect in irritable bowel syndrome.
Psychosom Med 2009; 71: 789–97.
55 Colloca L, Benedetti F. Placebos and painkillers: is mind as real as matter? Nat Rev Neurosci 2005; 6: 545–52.
81 Kaptchuk TJ, Goldman P, Stone DA, Stason WB. Do medical devices have enhanced placebo eff ects? J Clin Epidemiol 56 Levine JD, Gordon NC. Infl uence of the method of drug 2000; 53: 786–92.
administration on analgesic response. Nature 1984; 312: 755–56.
82 Kaptchuk TJ, Stason WB, Davis RB, et al. Sham device v inert pill: 57 Grevert P, Albert LH, Goldstein A. Partial antagonism of placebo randomised controlled trial of two placebo treatments. BMJ 2006; analgesia by naloxone. Pain 1983; 16: 129–43.
332: 391–97.
58 Fields HL, Levine JD. Placebo analgesia—a role for endorphins. 83 Kaptchuk TJ. The placebo eff ect in alternative medicine: can the Trends Neurosci 1984; 7: 271–73.
performance of a healing ritual have clinical signifi cance? 59 Benedetti F, Amanzio M, Maggi G. Potentiation of placebo Ann Intern Med 2002; 136: 817–25.
analgesia by proglumide. Lancet 1995; 346: 1231.
84 Kaptchuk T, Eisenberg DM. The persuasive appeal of alternative 60 Benedetti F, Amanzio M. The neurobiology of placebo analgesia: medicine. Ann Intern Med 1998; 129: 1061–65.
from endogenous opioids to cholecystokinin. Prog Neurobiol 1997; 85 Linde K, Streng A, Jurgens S, et al. Acupuncture for patients with 52: 109–25.
migraine: a randomized controlled trial. JAMA 2005; 61 Montgomery GH, Kirsch I. Mechanisms of placebo pain 293: 2118–25.
reduction. An imperical investigation. Psychol Sci 1996; 7: 174–75.
86 Melchart D, Streng A, Hoppe A, et al. Acupuncture in patients 62 Lipman JJ, Miller BE, Mays KS, Miller MN, North WC, Byrne WL. with tension-type headache: randomised controlled trial. BMJ Peak B endorphin concentration in cerebrospinal fl uid: reduced 2005; 331: 376–82.
in chronic pain patients and increased during the placebo 87 Brinkhaus B, Witt CM, Jena S, et al. Acupuncture in patients with response. Psychopharmacology 1990; 102: 112–16.
chronic low back pain: a randomized controlled trial. 63 Zubieta JK, Bueller JA, Jackson LR, et al. Placebo eff ects mediated Arch Intern Med 2006; 166: 450–57.
by endogenous opioid neurotransmission and μ-opioid receptors. 88 Haake M, Muller HH, Schade-Brittinger C, et al. German J Neurosci 2005; 25: 7754–62.
Acupuncture Trials (GERAC) for chronic low back pain: 64 Wager TD, Scott DJ, Zubieta JK, Wager TD, Scott DJ, Zubieta J-K. randomized, multicenter, blinded, parallel-group trial with Placebo eff ects on human mu-opioid activity during pain. 3 groups. Arch Intern Med 2007; 167: 1892–98.
Proc Natl Acad Sci USA 2007; 104: 11056–61.
89 Witt C, Brinkhaus B, Jena S, et al. Acupuncture in patients with 65 Wager TD, Rilling JK, Smith EE, et al. Placebo-induced changes osteoarthritis of the knee: a randomised trial. Lancet 2005; in fMRI in the anticipation and experience of pain. Science 2004; 366: 136–43.
303: 1162–66.
www.thelancet.com Vol 375 February 20, 2010
90 Cummings M. Modellvorhaben Akupunktur—a summary of the 99 Verne GN, Robinson ME, Vase L, et al. Reversal of visceral and ART, ARC and GERAC trials. Acupunct Med 2009; 27: 26–30.
cutaneous hyperalgesia by local rectal anesthesia in irritable 91 Linde K, Witt CM, Streng A, et al. The impact of patient bowel syndrome (IBS) patients. Pain 2003; 105: 223–30.
expectations on outcomes in four randomized controlled trials of 100 Gracely RH, Dubner R, Deeter WD, Wolskee PJ. Clinicians’ acupuncture in patients with chronic pain. Pain 2007; 128: 264–71.
expectations infl uence placebo analgesia. Lancet 1985; 325: 43.
92 Cherkin DC, Sherman KJ, Avins AL, et al. A randomized trial 101 Benedetti F, Arduino C, Costa S, et al. Loss of expectation-related comparing acupuncture, simulated acupuncture, and usual care mechanisms in Alzheimer’s disease makes analgesic therapies for chronic low back pain. Arch Intern Med 2009; 169: 858–66.
less eff ective. Pain 2006; 121: 133–44.
93 Sherman KJ, Cherkin DC, Ichikawa L, et al. Treatment 102 Miller FG, Wendler D, Swartzman LC. Deception in research on expectations and preferences as predictors of outcome in the placebo eff ect. PLoS Med 2005; 2: e262.
acupuncture for chronic back pain. Spine (in press).
103 Brody H. The lie that heals: the ethics of giving placebos. 94 Colloca L, Lopiano L, Lanotte M, Benedetti F. Overt versus covert Ann Intern Med 1982; 97: 112–18.
treatment for pain, anxiety, and Parkinson’s disease. Lancet Neurol 104 Hrobjartsson A, Norup M, Hrobjartsson A, Norup M. The use of 2004; 3: 679–84.
placebo interventions in medical practice—a national 95 Price DD. Assessing placebo eff ects without placebo groups: an questionnaire survey of Danish clinicians. Eval Health Prof 2003; untapped possibility? Pain 2001; 90: 201–03.
26: 153–65.
96 Amanzio M, Pollo A, Maggi G, Benedetti F. Response variability 105 Tilburt JC, Emanuel EJ, Kaptchuk TJ, et al. Prescribing “placebo to analgesics: a role for non-specifi c activation of endogenous treatments”: results of national survey of US internists and opioids. Pain 2001; 90: 205–15.
rheumatologists. BMJ 2008; 337: a1938.
97 Benedetti F, Maggi G, Lopiano L, et al. Open versus hidden 106 Park LC, Covi L. Nonblind placebo trial: an exploration of neurotic medical treatments: the patient’s knowledge about a therapy patients’ responses to placebo when its inert content is disclosed. aff ects the therapy outcome. Prevent Treat 2003; 6: ArtID 1a.
Arch Gen Psychiatry 1965; 12: 336–45.
98 Pollo A, Amanzio M, Arslanian A, Casadio C, Maggi G, cacy of a non blind placebo prescription. Benedetti F. Response expectancies in placebo analgesia and their Encephale 2003; 29: 68–71.
clinical relevance. Pain 2001; 93: 77–84.
www.thelancet.com Vol 375 February 20, 2010

Source: http://www.fisica.uh.cu/rationalis/etica-placebo/Lancet2010/Lancet2010.pdf

FinanÇas pÚblicas e orÇamento participativo: a experiÊncia municipal brasileira

V Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública, Santo Domingo, 24-27 Oct. 2000 Finanças públicas e orçamento participativo: a experiência municipal brasileira Eduardo de Lima Caldas Curso de Mestrado em Administração Pública e Governo Introdução O objetivo deste trabalho é apresentar a experiências de elaboração de orçamen

Untitled

POWER FOR PROFESSIONALS Internationales Magazin für Werbeartikel MAI 2012 51. Jahrgang Ralf Rühlemann Kasper & Richter „Alles muß Sinn machen“ Yasmin von Schweinitz Lunari Vielseitige Innovationen Werbeartikel-Guide Taschen, Leder, Reise, Gesundheit, Fitness Lexon Kunst für jeden Tag mt products Die besondere Note Scholz Promotion

Copyright © 2010-2014 Medical Articles